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SUMMARY 


A  descriptive  and  analytical  investigation  of  the  training  information 
and  evaluation  system  which  supports  initial  qualification,  pilot/navigator- 
upgrade,  and  requalification  training  for  8-62  and  KC-135  aircrews  at  the 
93  Bombardment  Wing,  Castle  AF8,  California,  was  conducted.  The  study  was 
envisaged  as  a  baseline  from  which  to  develop  eventual  improvements  to 
evaluation  practice.  A  description  of  the  organization  and  functioning  of  the 
information  and  evaluation  system  is  presented,  along  with  a  synopsis  of 
the  system  which  is  grouped  according  to  the  informational  categories  of 
student  evaluation,  program  evaluation,  and  management  of  student  instruction. 
It  was  concluded  that  ample  provisions  exist  for  evaluating  students  as  they 
progress  through  the  program  of  instruction,  yet  most  of  this  information  is 
not  used,  in  turn,  for  systematic  training  program  evaluation.  The 
limitations  of  the  current  information  and  evaluation  system  were  viewed  as  a 
function  of  past  Air  Force  requirements  for  evaluation,  the  current  manual 
information  system  which  renders  routine  information  collection  and  analysis 
impractical,  parallel  evaluation  functions  being  performed  by  several  offices 
within  the  Wing,  and  the  need  for  an  overall,  integrated  evaluation  plan. 
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PREFACE 


This  work  was  performed  in  support  of  AFHRL  Work  Unit  No.  1123-03-83, 
Flying  Training  Research  Support.  The  report  documents  current  evaluation 
practices  in  formal  school  training  for  B-52  and  KC-135  aircrews.  The  study 
is  part  of  a  larger  effort  in  which  evaluation  for  all  phases  of  B-52  and 
KC-135  training  is  described  in  order  to  provide  a  baseline  from  which  to 
design  improvements  to  the  evaluation  of  aircrew  training.  This  author  wishes 
to  acknowledge  the  contributions  of  several  individuals.  Dr.  Robert 
Nullmeyer,  AFHRL/OT,  first  suggested  the  need  for  a  study  of  current  aircrew 
training  evaluation  and  provided  helpful  comments  as  the  paper  was  being 
drafted.  Separate  reviews  of  a  draft  manuscript  were  also  provided  by  Dr. 
Marty  Rockway,  UDRI;  Dr.  Thomas  Killion,  AFHRL/OT;  and  Majs  Charles 
Wennermark  and  Stewart  Monti,  93  6MW/D05.  This  author  accepts,  however,  sole 
responsibility  for  the  conclusions  contained  in  the  report.  Most  importantly, 
this  author  wishes  to  extend  a  note  of  thanks  to  the  men  and  women  of  the  93 
Bombardment  Wing,  who  graciously  consented  to  interviews  about  their  training 
information  system  and  made  numerous  documents  and  other  materials  readily 
available. 
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AIRCREW  TRAINING  EVALUATION: 

B-t)2  AND  KC-13b  FORMAL  SCHOOL  TRAINING 


I.  INTRODUCTION 

This  report  documents  a  descriptive  and  analytical  investigation  of  the 
information  system  which  supports  the  initial  qualification,  pilot^avigator- 
upgtade,  and  requa’ification  training  programs  for  B-52  and  KC-135  aircrews  at 
the  93  Bombardment  Wing  (BMW),  Castle  Air  Force  Base,  California.  The  current 
version  of  the  information  system  is  largely  a  manual  form  of  record  keeping, 
which  provides  data  used  in  the  management  and  evaluation  of  the  aircrew 
training  programs.  While  both  of  these  functions  are  addressed  in  the  present 
study,  evaluation  is  of  primary  interest. 

Several  recent  developments  are  likely  to  increase  future  requirements 
for  the  evaluation  of  aircrew  training  systems,  or  at  least  make  evaluation 
more  visible.  For  example.  Department  of  Defense  Directive  No.  1430.13, 
Training  Simulators  and  Devils  (Office  of  Secretary  of  Defense,  22  August 
1986)  specifies  that  training  effectiveness  evaluations  are  to  be  conducted  to 
ensure  that  training  devices  meet  training  requirements  and  effectiveness 
levels.  A  trend  toward  contracted  aircrew  training--with  C-5,  E-3A,  KC-10, 
and  C-130  providing  examples--also  accentuates  the  importance  of  evaluation, 
since  organizations  other  than  the  military  are  conducting  large  portions  of 
the  training.  Compliance  with  contractual  requirements  is  a  relevant 
evaluative  issue  in  contracted  aircrew  training.  The  importance  of  a 
comprehensive  evaluation  of  training  was  recognized  early  in  the  C-130 
program,  and  this  led  to  the  formulation  of  a  test  and  evaluation  plan  during 
the  front-end  analysis  (see  Fishburne,  Williams,  Chatt,  &  Spears,  1987, 
pp.  29-40;  Spears,  1986)  and  the  subsequent  requirement  for  a  training  system 
test  and  evaluation  plan  in  the  Statement  of  Work  for  the  C-130  Aircrew 
Training  System  (Aeronautical  Systems  Division,  1  June  1986). 

Economic  constraints  produce  additional  pressures  to  conduct  evaluations 
of  training,  as  competing  demands  are  placed  on  limited  budgets.  Accordingly, 
formal  demonstrations  of  effectiveness  are  especially  critical  with  respect  to 
expensive  program  resources  such  as  weapon  system  trainers  and  flying  hours. 

In  this  regard,  the  General  Accounting  Office  (GAO),  in  their  review  of 
Tactical  Air  Command  (TAC)  and  Strategic  Air  Cotnmand  (SAC)  flying  hour 
programs  (GAO,  1986),  concluded  that  justification  of  the  flying  hour  budgets 
submitted  to  the  Congress  had  not  been  based  on  documented  demonstrations  that 
increased  flying  hours  lead  to  increases  in  the  combat  capability  of 
aircrews.  The  GAO  also  argued  that  the  relationship  between  flying  hours  and 
combat  capability  should  be  based  on  objective  evidence,  not  judgment  alone. 
More  generally,  the  Defense  Science  Board  (1982)  concluded  that  information  to 
support  management  decisions  in  military  training  is  sparse. 

Collectively,  these  factors  suggest  that  evaluation  of  aircrew  training 
is  a  serious  concern,  and  that  it  must  be  practiced  in  a  formal,  systematic 
fashion.  Only  this  approach  to  evaluation  is  capable  of  satisfying  the 
increasingly  stringent  requirements  for  information  concerning  the 
effectiveness  of  complex  training  systems,  including  expensive  components  such 
as  weapon  system  trainers  and  flying  hours.  This  is,  indeed,  the  perspective 
adopted  in  the  present  study.  In  this  regard,  it  is  noteworthy  that  credible 
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evaluations  demonstrating  the  effectiveness  of  programs  in  other  settings  have 
often  resulted  in  the  maintenance  of  those  programs;  in  some  cases,  they  have 
resulted  in  program  expansions--even  in  times  of  scarce  monetary  resources 
(Wholey,  1986).  The  objective,  then,  is  to  make  aircrew  training  evaluation 
more  responsive  to  the  requirements  imposed  upon  it. 

In  a  fundamental  sense,  to  determine  that  a  program  or  training  regimen 
is  effective  requires  an  assessment  of  outcomes;  a  program  is  judged  effective 
to  the  extent  that  desired  outcomes  are  actually  attained  (Patton,  1986, 
p.  345;  Scriven,  1982,  p.  48).  In  aircrew  training,  such  outcomes  typically 
include  measures  of  aircrew  learning  and  performance.  It  is  often  necessary, 
however,  to  understand  why  or  how  a  program  is  effective  or  ineffective;  for 
this,  a  separate  evaluation  is  required.  This  evaluation  has  been  termed  a 
"process  evaluation"  (Judd,  1987;  Patton,  1986;  Scriven,  1982,  p.  121). 

Process  evaluations  provide  detailed  knowledge  about  program  implementation 
and  operations,  and  unintended  effects  are  often  detected  from  them. 

Information  from  process  evaluations  is  important,  particularly  if  one  wishes 
to  improve  program  effectiveness.  Some  evaluators  have  argued  that  program 
improvement  is  the  most  important  function  of  evaluation  (Cronbach,  1963, 

1982;  Stuff lebeam,  1985).  Recently,  evaluators  have  prudently  suggested  that 
process  and  outcome  evaluations  be  combined  in  assessments  of  programs 
(Cronbach,  1982;  Judd,  1987),  although  the  past  tendency  has  been  to  focus  on 
outcomes  alone. 

These  basic  types  of  evaluation,  if  conducted  on  a  routine  basis,  should 
provide  much  of  the  necessary  information  from  which  to  determine  the 
effectiveness  and  efficiency  of  aircrew  training.  If  used  properly,  they 
should  also  furnish  information  from  which  to  improve  training  and  justify 
particular  uses  of  training  resources.  A  review  of  selected  B-52  and  KC-135 
operational  training  programs  suggests,  however,  that  evaluation  is  focused 
primarily  on  the  performance  of  individuals  or  aircrews,  not  necessarily  on 
the  effectiveness  and  efficiency  of  training  (Bruce,  Rockway,  Povenmire,  & 
Killion,  in  preparation).  A  similar  conclusion  was  asserted  in  a  recent 
review  of  performance  measurement  requirements  in  TAG  training  (Waag,  Pierce, 

&  Fessler,  1987).  This  emphasis  is  easily  understood  in  the  context  of  the 
military,  as  it  is  individuals  and  aircrews  which  must  ultimately  perform 
combat  missions,  and  information  which  assists  in  making  judgments  of  their 
capabilities  is  vital.  Yet,  this  emphasis  may  also  lead  to  the  neglect  of 
evaluating  important  aspects  of  the  training  system,  the  means  by  which 
combat-capable  aircrews  are  produced. 

The  findings  of  Bruce  et  al.  (in  preparation)  and  Waag  et  a1.  (1987) 
serve  to  illustrate  that  evaluation  can  have  multiple  focuses.  Evaluating 
selected  aspects  of  a  training  system  can  be  construed  as  a  form  of  contingency 
management;  using  information  to  monitor  the  performance  of  a  feature  of  the 
training  system  may  actually  increase  the  likelihood  that  training  will  be 
responsive  in  prescribed  ways  (Bruce,  Nullmeyer,  &  Rockway,  1987).  For 
example,  if  it  is  important  that  a  training  system  be  rapidly  updated  because 
of  changes  in  the  aircraft  mission  or  equipment,  an  informational  mechanism 
can  be  developed  which  targets  ongoing  and  projected  changes,  how  changes  were 
implemented,  and  the  amount  of  time  to  complete  actual  changes  to  the  training 
system.  Such  a  method  would  likely  increase  the  probability  that  the  training 
system  would  incorporate  needed  changes  relatively  quickly.  It  is  prudent, 
therefore,  to  specify  as  comprehensively  as  possible  the  critical  focuses  of 


an  evaluation,  which  may  include  program  recipients,  resources,  and  training 
strategies.  A  related  point  is:  Just  as  evaluation  has  multiple  focuses,  the 
same  information  can  serve  multiple  functions.  For  instance,  performance 
assessments  of  individuals  can  often  be  aggregated  and  analyzed  in  diverse 
ways  to  provide  information  about  program  outcomes.  These  considerations  are 
intended  to  serve  as  important  background  elements  as  we  now  examine 
information  collection  and  use  at  the  93  BMW. 


II.  OBJECTIVES 

The  present  investigation  is  part  of  a  larger  effort  in  aircrew  training 
evaluation,  which  will  include  an  examination  of  information  collection  and 
evaluation  practice  in  initial  qualification,  mission  qualification,  and 
continuation  training  for  B-52  and  KC-135  aircrews.  The  reason  for  the  study 
was  simple:  Improvetnents  in  aircrew  training  evaluation  must  oe  based  upon  an 
adequate  understanding  of  current  practice.  Accordingly,  tnis  work  is 
intended  to  serve  as  a  baseline  for  devising  eventual  improvements  in  aircrew 
training  evaluation  within  a  formal  school  setting  of  SAC.  The  objectives  of 
the  effort  were:  (a)  to  identify  the  main  items  of  information  collected  on 
students  as  they  progress  through  the  initial  qualification,  pilot/navigator- 
upgrade,  and  requalification  phases  of  training;  (b)  to  identify  additional 
information  which  is  collected  for  evaluation  and  improvement  of  the  programs; 
(c)  to  identify  relevant  data  collected  for  the  management  of  student 
instruction  and  the  operation  of  the  training  system;  (d)  to  determine  how 
collected  information  is  actually  used;  (e)  to  identify  shortfalls  in  the 
evaluation  process;  and  (f)  to  suggest  some  initial  improvements  to  evaluation 
practice,  within  the  constraints  of  the  current  information  system.  This 
investigation  does  not  include  the  formulation  of  a  comprehensive  information 
and  evaluation  system  for  the  93  BMW,  as  that  would  involve  a  more  extensive 
development  process.  It  is  considered,  however,  as  a  first  step  in  that 
endeavor. 


III.  METHODS  OF  INFORMATION  COLLECTION 

Interviews  were  conducted  with  33  individuals  from  the  following  Wing 
offices:  329  Combat  Crew  Training  School  (CCTS),  the  93  Air  Refueling  Squadron 
(AREFS),  the  328  Bombardment  Squadron  (BMS),  the  Standardization/Evaluation 
Division  (DOVB/K),  B-52  and  KC-135  Training  Program  Management  (D02B/K),  the 
Operations  Systems  Management  Branch  (uOTF),  the  Instructional  Systems 
Development  Branch  (DOb),  and  Wing  Scheduling  (DOT).  Individuals  interviewed 
included  training  managers,  curriculum  developers  and  evaluators.  Wing 
evaluators,  academic-training  section  heads  and  the  Director  of  Academics, 
flying-squadron  operations  officers  and  flight  commanders.  Wing  schedulers, 
and  clerical  personnel.  The  interviews  were  structured  to  identify  all 
training  information  collection,  use,  and  dissemination.  Follow-up  interviews 
were  conducted  by  telephone  or  in  person  with  different  personnel,  to  verify 
training  database  inputs  and  gather  additional  information  about  the  structure 
and  functioning  of  the  system.  Samples  of  all  data  forms  and  charts  were 
obtained  from  each  office  for  examination.  A  Mission  Review  Panel  was 
attended  to  observe  how  flying  training  events  were  documented  and  verified, 
and  to  identify  interfaces  with  other  components  of  the  aircrew  training 
system  such  as  scheduling  and  maintenance. 
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Samples  of  the  following  reports  were  reviewed;  CCTS  Graduate  Summaries 
prepared  by  D02B/K,  which  include  Standardization/Evaluation  (S/E)  results  and 
analyses;  reports  of  needs-analysis  visits,  as  specified  in  APR  50-38  and  SACR 
55-70;  Curriculum  Review  Group  (CRG)  and  Curriculum  Review  Board  (CRB) 
minutes;  and  Training  Review  Panel  (TRP)  reports.  These  were  examined  to 
determine  what  training  information  is  routinely  reported,  reviewed,  and  acted 
upon,  and  then  disseminated  within  and  outside  the  Winy.  Several  CRG  and  CRB 
meetings  had  been  attended  on  previous  occasions. 

Applicable  regulations,  manuals,  pamphlets,  and  operating  instructions 
were  identified  to  assist  in  describing  and  understanding  the  y3  BHW  training 
information  system.  In  addition,  a  current  organizational  chart.  Deputy 
Commander  for  Operations,  was  used  to  identify  organizational  components  of 
the  training  system  and  the  chain  of  command. 


IV.  INFORMATION  SYSTEM  DESCRIPTION 

The  results  of  this  inquiry  are  organized  along  two  dimensions.  First, 
each  office  within  the  93  BMW  which  collects,  processes,  and  uses  training 
information  is  listed,  and  the  information  collected  and  used  in  each  of  them 
is  then  described.  Second,  there  is  an  attempt  to  preserve  the  temporal 
sequence  of  information  collection.  For  example.  Flight  Records  (DOTF) 
receives  training  information  from  Air  Training  Command  (ATC)  directly,  or 
this  information  is  hand-carried  to  the  93  BMW  by  the  incoming  student.  This 
is  considered  the  first  informational  input  into  the  aircrew  training 
database.  As  students  progress  through  the  training  curriculum,  the  academic 
and  flying  squadrons  collect  training  information,  in  sequence.  Eventually, 
graduates  and  gaining-unit  squadron  commanders  return  external  evaluation 
questionnaires,  the  last  item  of  information  in  the  temporal  sequence,  to  the 
93  BMW. 

The  CRG,  CRB,  and  TRP  are  formal  bodies  within  the  93  BMW  which  review 
programs  and  make  decisions.  The  information  which  is  passed  up  through  the 
chain  of  command  and  typically  reviewed  by  these  bodies  is  also  presented.  In 
addition,  information  which  is  disseminated  outside  the  93  BMW  is  described. 


Systems  of  Record  Keeping 

Five  systems  of  student  record  keeping  were  identified:  Personnel 
Records  (329  CCTS/DA),  Flight  Records  (UUTF),  Training  Records  (329  CCTS, 

93  AREFS,  and  328  BMS),  Standardization/Evaluation  Records  (DOVB/K),  and 
External  Evaluation  Records  (005).  Personnel  records  will  not  be  treated  in 
this  report.  Some  interfaces  between  record  keeping  systems  will  be 
identified  below. 


Operations  Systems  Management  Branch,  or  Flight  Records  (DOTF) 

Student  records  arriving  from  Undergraduate  Pilot  Training  (UPT)  or 
Undergraduate  Navigator  Training  (UNT)  are  first  processed  at  DOTF.  The 
Flight  Record  Folders  and  Flight  Evaluation  Folders  from  UPT  and  UNT  are  kept 
on  file  until  the  students  complete  the  CCTS  training  program.  These  folders 


4 


are  also  retained  in  DOTF  for  those  students  undergoing  upgrade  and 
requalification  training,  p’light  Record  Folders  contain  the  Individual  Flying 
Record  and  the  Flying  History  Report.  The  Individual  Flying  Record  contains  a 
running  nistory  of  flying  hours  by  sortie,  and  each  sortie  is  separated 
according  to  the  type  of  aircraft.  The  Flying  History  Report  contains  flying 
hour  totals  for  each  aircraft  in  a  given  duty  position,  and  it  includes  career 
totals  for  all  flying,  including  simulators.  Flight  Evaluation  Folders 
contain  AF  Form  8,  Certificate  of  Aircrew  Qual if i cat  ion;  ATC  Form  1122, 

Summary  Performance  Report,  which  is  a  brief  descriptive  summary  of 
performance;  and  A^C  Form  240-5,  Summary  Record  of  Training,  which  contains 
courses,  hours,  and  grades  in  each  course.  These  records  are  available  for 
examination  by  all  training  squadrons,  but  they  are  not  routinely  used  by  all 
instructor  personnel.  Some  academic  sections  have  made  independent 
arrangements  with  UPT  and  UHT  to  obtain  copies  of  these  records,  but  they  are 
not  received  on  a  reliable  basis. 

Crew  Mission  Accomplishment  Reports  (MARs)  are  completed  at  the 
conclusion  of  each  sortie  for  student  crews  and  permanent  party  crews  (e.g., 
Wing  instructor/evaluator  crews  and  all  craws  from  the  924  AREFS).  Crew  MARs 
are  used  primarily  for  tracking  fuel  consumption,  flying  time,  and  number  of 
sorties  scheduled/flown.  Individual  MARs  are  completed  for  permanent  party 
personnel  only.  Individual  MARs  are  not  completed  for  students  undergoing 
initial  qualification  training,  as  these  students  are  not  yet  qualified  in 
their  primary  aircraft.  Individual  MARs  are  also  not  completed  for  upgrade  or 
requalification  students.  After  verification  in  Mission  Review  Panel,  all 
MARs  are  entered  into  the  Air  Force  Operational  Resource  Management  System 
(AFORMS)  computer,  but  only  permanent-party  individual  MARs  which  record 
flying  events  accomplished  are  actually  tracked  in  this  system.  Flying  events 
accomplished  by  those  undergoing  training  in  the  initial  qualification  program 
are  not  tracked  by  AFORMS,  but  their  total  flying  time  logged  in  the  aircraft 
is  tracked  by  this  system. 


329  Combat  Crew  Training  School 

Academic  sections  are  organized  according  to  crew  position,  such  as 
pilot/copilot  and  navigator,  for  both  the  B-52  and  the  KC-135.  Although  there 
is  some  variation  between  the  academic  sections  in  the  specific  items  of 
information  recorded  as  students  progress  through  instruction,  there  are  some 
general  trends.  Each  section  records  examination  scores  on  a  separate  form 
for  each  student.  The  Appendix  to  the  present  report  contains  an  example  of 
one  such  form,  the  Initial  Qualification  Criterion  Test  Record,  which  is  used 
in  B-52  pilot  academics  (see  Appendix,  p.  28).  Scores  may  be  expressed  as 
pass-fail  or  percentage  correct.  End-of-course  examination  scores  are  also 
recorded  on  these  forms.  In  addition,  each  section  records  accomplishments 
and/or  performance  ratings  for  each  session  with  aircrew  training  devices. 

Some  sections  record  the  amount  of  time  each  student  spends  on  each  unit  of 
instruction  and  the  date  of  completion. 

Data  on  student  learning  and  performance  are  used  for  the  day-to-day 
management  of  instruction  and  the  evaluation  and  tracking  of  each  student's 
progress.  These  data  may  also  be  used  for  within-section  program  monitoring. 
They  are  not  processed,  analyzed,  and  disseminated  for  routine  evaluation  of 
the  academic  program.  They  remain  in  section  files  at  the  329  CCTS  after 
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instruction  is  completed.  Records  are  retained  from  6  months  to  J  years, 
depending  on  the  section. 


At  the  conclusion  of  academic  instruction,  each  student  completes  an 
evaluation  on  Castle  Form  49,  CCTS  Student  Critique:  Academics  Phase' 

(see  Appendix,  p.  29).  Students  a’^e  requested  to  rate  instructors,  individual 
assistance,  training  methods,  ti^aining  literature,  visual  aids,  synthetic 
training  aids  (aircrew  training  devices),  examinations,  and  the  overall 
course.  The  ratings  are  either  outstanding,  satisfactory,  c"  unsatisfactory; 
and  comments  or  recommendaticns  for  improvement  are  to  be  provided  if  an  area 
has  been  rated  unsatisfactory.  In  essence,  this  information  is  concerned  with 
the  processes  of  instruction;  for  example,  whether  instructors  were 
understandable  and  helpful,  the  usefulness  of  visual  aids,  the 
understandabil ity  of  examinations,  and  the  use  of  training  time.  Course 
critiques  are  typically  circulated  witnin  earh  academic  section  for  review 
and/or  action  by  instructors  and  section  heuds. 

All  course  critiques  are  forwarded  to  the  Director  of  Academics  for 
"recapitulation,"  and  the  results  are  summarized  by  class.  The  recapitulations 
contain  the  overall  frequencies  of  outstanding,  satisfactory,  and 
unsatisfactory  ratings  for  each  of  the  categories  listed  above,  and  critique 
items  are  also  enumerated  by  category.  The  recapitulation  is  disseminated  to 
every  major  training  section  within  the  93  BMW  for  review,  comment,  and/or 
action.  Appropriate  training  managers  throughout  the  Wing  must  respond  to 
unfavorable  critique  items.  This  process  appears  to  serve  as  a  check  on 
unwarranted  student  comments  or  a  means  of  corroborating  deficiencies  which 
must  then  be  corrected. 

At  the  completion  of  academic  instruction,  each  student  is  assigned 
distinguished  graduate  (DG)  points,  and  Castle  Form  219,  Education/Training 
Report,  is  completed  by  the  academic  instructor.  This  1 atter  form  pertains  to 
general  attitude,  conduct,  and  fitness.  These  items  of  information  are  then 
sent  to  the  flying  squadrons.  CAFBR  50-b,  Section  l.b.,  states  that  each 
academic  section  will  provide  a  summary  of  academic  training  and  exaiiiination 
results  to  the  flying  squadrons,  presumably  to  assist  the  flying  squadrons  in 
identifying  strengths  and  weaknesses  of  students.  In  practice,  this  does  not 
occur. 


Each  flying  squadron  constructs  training  folders  which  cont.’tn  records 
for  the  flightline  phase  of  instruction  for  each  student.  The  main  items 
consist  of  Progress  Reports  (PRs),  one  of  which  is  completed  for  each  training 
session  in  a  weapon  system  traintr  (WST)  or  aircraft  sortie,  and  the  master 
Training  Accomplishment  Report  (TAR).  Training  events  accomplished  during 


^Student  critiques  are  being  revised  by  the  DOS.  The  new  critiques  will 
include  an  expanded  informational  format,  direct  questions  with  an  associated 
rating  scale,  and  they  will  be  processed  by  a  computer. 
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flightline  instruction  are  recorded  on  the  TAR.  Records  are  also  kept  for 
training  received  in  cockpit  procedures  trainers  (CRTs),  T-lOs,  T-4s,  and 
other  aircrew  training  devices.  Some  of  these  records,  however,  are  not 
contained  in  the  main  training  folders.  Instead,  they  may  be  kept  in 
facilities  which  house  particular  aircrew  training  devices. 

Instructors  enter  items  of  information  on  the  above  forms  after  each 
instructional  session  in  an  aircrew  training  device  or  the  aircraft. 
Proficiency  ratings  are  entered  on  PRs  (see  Appendix,  p.  30)  for  each 
repetition  of  a  particular  flying  event  in  the  WST  or  aircraft.  Proficiency 
ratings  are  assigned  according  to  the  criteria  of  a  7-point  grading  scale  in 
use  at  the  CCTS,  which  ranges  from  1. 0-4.0  (including  half-points),  with  3.0 
considered  as  "initial  proficiency."  Interestingly,  ratings  for  each 
accomplished  flying  event  on  the  PR  are  not  arranged  sequentially;  rather,  the 
record  contains  a  mere  tally  of  the  number  of  times  an  event  was  performed  at 
a  given  level  of  proficiency.  Accordingly,  precise  sequential  information 
cannot  be  derived  for  tracking  student  progress.  Space  is  provided  on  the  PR 
for  descriptions,  discrepancies,  and  critiques  of  student  performance. 
Instructors  enter  the  number  of  repetitions  for  each  flying  event  accomplished 
during  each  aircraft  sortie  on  the  TAR  (see  Appendix,  p.  31).  When  proficient 
performance  for  an  event  has  been  achieved  by  the  student,  a  circle  is  drawn 
around  the  corresponding  entries  for  that  particular  sortie.  Proficiency 
levels,  grades,  or  training  accomplishments  are  entered  for  sessions  with 
other  aircrew  training  devices  such  as  T-lOs,  T-4s,  and  CRTs. 

After  each  flight,  students  and  instructors  verify  the  training 
accomplished.  The  TARs  and  PRs  are  also  reviewed  by  flight  commanders  after 
each  sortie,  and  the  TAR  becomes  part  of  the  package  which  is  submitted  to  the 
Mission  Review  Panel  the  day  after  the  flight.  If  a  scheduled  training  event 
is  not  accomplished  on  a  given  sortie,  Castle  Form  144,  Student  Action  Record, 
is  completed.  It  is  forwarded  to  D02B/K  via  Mission  Review  Panel  for  action, 
which  may  include  rescheduling  of  the  particular  event(s). 

TARs  are  reviewed  by  flight  commanders  prior  to  each  student's  last 
flight  before  the  SACR  60-4  checkride.  This  is  done  to  ensure  that  training 
requirements  are  completed,  or  to  notify  D0TB/D02B  or  D0TK/D02K  of  incomplete 
items  so  that  additional  training  may  be  scheduled  or  event  waivers  obtained. 
Copies  of  TARs  and  PRs  are  prepared  by  instructors,  and  these  and  the  master 
sets  are  signed  by  flight  commanders  and  D02B  or  D02K.  These  copies  are 
eventually  hand-carried  by  graduates  to  their  gaining  units  after  completion 
of  the  CCTS  program.  Master  copies  are  retained  by  the  flying  squadrons  for 
1  year.  CAFBR  50-5  states  that  if  a  Follow-On  Operational  Test  and  Evaluation 
(FOT&E)  of  an  aircrew  training  device  is  in  effect,  copies  of  PRs  are  to  be 
sent  to  the  FOT&E  Test  Director  at  the  005.  The  reports  are  then  to  be 
returned  to  the  flying  squadrons  for  retention.  TARs  and  PRs  are  used  only  to 
track  event  accomplishment  and  to  monitor  and  evaluate  student  progress  or 
proficiency  throughout  the  flying  phase  of  training.  They  are  not  used  to 
conduct  a  formal  evaluation  of  flying  training. 

At  the  conclusion  of  the  flighcline  phase  of  training,  students  complete 
Castle  Form  55,  CCTS  Student  Critique:  Flying  Phase  (see  Appendix,  p.  32), 
which  contains  the  evaluative  categories  of  instructors,  flying  training, 
training  aids,  Standardization/Evaluation  Division,  and  overall  evaluation  of 
training.  These  critiques  are  reviewed  by  flight  commanders  and  squadron 
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operations  officers.  They  are  then  disseminated  for  review  within  the  Wing. 
The  DG  points  are  assigned  to  students  by  the  squadrons,  and  together  with 
Castle  Form  219,  they  are  then  forwarded  to  329  CCTS/UA  for  final  processing. 


Standardization/Evaluation  Division  (DOVB/K) 


All  procedures  for  the  S/E  program  are  contained  in  the  SACR  60-4 
Volumes.  DOVB/K  administer  written  examinations  over  such  content  areas  as 
aircraft  procedures,  aircraft  performance  data.  Air  Force  and  SAC  regulations, 
and  emergency  procedures.  Additional  areas  may  be  assessed,  depending  upon 
crew  position.  Some  examinations  are  open-book,  while  others  are  closed- 
book.  All  examinations  must  oe  completed  and  passed  prior  to  the  SACR  bU-4 
checkride.  The  examination  scores  remain  on  file  for  1  year.  CRT  checks  for 
bomber  and  tanker  pilots,  evaluations  in  celestial  training  devices  for  bomber 
navigators,  and  T-4  checks  for  electronic  warfare  officers  are  also  conducted 
prior  to  checkrides.  Some  of  these  checks  may  be  conducted  by  the  instructors, 
instead  of  DOV  personnel. 

Tanker  and  bomoer  pilots  are  administered  instrument  checks  as  part  of 
their  SACR  60-4  evaluation.  Pilots  are  also  evaluated  using  the  Fuel 
Conservation  Critique  Form.  Upon  completion  of  checkrides,  DOVB/K  personnel 
fill  out  aF  Form  8,  CertTficate  of  Aircrew  Qualification  (see  Appendix, 
p.  33),  for  each  individual .  This  form  includes  the  assignment  of  grades  for 
ground  and  flight  phases  of  the  check.  Evaluators  also  assign  a  qualification 
level--Ql,  Q2,  or  Q3--the  latter  of  which  corresponds  to  the  unqualified  level. 
DOV  Form  FL  #16,  Corrective  Training,  is  completed  for  those  individuals  who 
require  correct i ve  training.  This  form  has  sections  for  recording 
discrepancies  and  recommended  actions.  DOV  Form  FL  #20,  Notification  of 
Corrective  Action,  is  also  used  for  corrective  action  but  typically  refers  to 
ground  training.  Trends  analysis  worksheets  are  completed  for  each  individual 
after  the  checkride.  Each  student  is  also  assigned  DG  points  at  the 
conclusion  of  all  qualification  activity. 

All  AF  Form  8s  are  reviewed  by  the  DOV  branch  chiefs  and  sent  to  the 
appropriate  flight  commanders  in  the  flying  squadrons  for  subsequent  review. 

AF  Form  8s  are  then  returned  to  the  DOV.  Individual  AF  Form  8s  are  sent  to 
the  graduate's  gaining  unit.  When  the  graduate  arrives  at  a  gaining  unit,  the 
Wing  DOV  arranges  for  a  joint  review  and  signature  of  the  AF  Form  8. 

Checkride  results  for  individuals  (CAFB  Form  74,  Initial  Qualification  Check 
Results)  and  each  class  (DOV  FL  #12,  Stan  Eval  Report)  are  forwarded  to 
001/2.  CAFB  Form  74  includes  the  qualification  level  and  explanations  for 
less-than-qual if ied  activity.  Class  results  on  DOV  FL  #12  (see  Appendix 
p.  34)  are  expressed  as  overall  numbers  in  each  qualification  category  and 
percentages  of  students  in  each.  Annual  class  results  which  are  pooled  across 
classes  are  also  tracked  on  this  form,  presumaoly  to  provide  a  baseline  for 
comparison  with  results  of  each  individual  class. 

The  trends  analysis  program  is  designed  as  a  tool  for  improving  the 
aircrew  training  program.  Negative  trends  in  ground  or  flying  activity  may 
result  from  observed  deficiencies  in  knowledge  or  performance,  and  they  may 
indicate  that  training  requires  revision  or  restructuring.  Combined  results 
of  individual  SACR  60-4  evaluations,  the  raw  data  of  which  appear  on  the 
Trends  Analysis  Worksheet  (see  Appendix,  p.  35),  form  the  basis  for  trends 
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analysis  within  the  Wing.  Ratings  on  these  worksneets  are  divided  into  five 
different  categories.  The  first  four  categories  correspond  to  ascending 
levels  of  qualification  and  the  fifth  indicates  that  an  activity  was  not 
observed  on  a  particular  check.  Examinees  are  rated  in  numerous  areas  of 
flying  activity  such  as  communications,  crew  coordination,  navigation,  and  air 
refueling.  Trends  analysis  results  are  entered  into  a  computer  at  the  DOV, 
and  the  perf onnanLe  of  eacn  class  of  students,  by  crew  position,  is  compared 
to  a  cumulative  datauase  composed  of  trends  analysis  results  from  previous 
classes.  Froin  these  comparisons,  trends  are  detected  over  time.  Deviations 
below  the  "norm"  are  first  reported  as  "areas  of  interest."  If  such 
deviations  continue  over  2  to  3  classes,  they  are  reported  as  "trends." 

Trends  require  corrective  action,  but  there  is  no  centralized  focal  point 
within  the  organization  foi'  confirming  the  validity  ana  identifying  the  source 
of  trends,  or  for  ensuring  their  subsequent  diagnosis  and  correction.  Trends 
are  reported  at  monthly  CRH,  CRB,  and  TRP  meetings,  and  action  may  be  taken 
upon  them  in  these  forums.  Identified  trends  may  also  be  the  subject{s)  of 
discussion  at  instructor  meetings,  at  which  time  they  are  noted  and  become  the 
object  of  "special  attention"  in  suosequent  instruction  with  students.  As 
such,  they  are  acted  upon  prescr ipti vely,  outside  the  purview  of  a  formalized 
process  of  detection,  validation,  diagnosis,  and  correction. 

Monthly  compilations  of  individual  5ACR  dO-4  evaluations,  subdivided 
according  to  ground  and  in-flight  areas  of  activity  which  are  the  same  as 
those  for  trends  analysis,  are  sent  to  the  1st  Combat  Evaluation  Group, 
Barksdale  AFB,  Louisiana,  on  SAC  Form  111,  Standardization  Data  Transcript. 
These  monthly  compilations  become  part  of  the  SAC-DOT(M)  7109  report  (see  bACR 
60-4,  Vol.  I),  and  tney  form  the  basis  for  the  Command  trends  analysis  program. 

DOV  is  also  involved  in  other  evaluative  activities.  These  include 
no-notice  evaluations  of  classes,  trainers,  briefings,  or  critiques,  which  are 
recorded  on  Castle  Form  11,  Academic  Evaluation  Worksheet  (see  Appendix, 
p.  36)  and  DOV  FL  #6,  Training" Device  Evaluation  (see  Appendix,  p.  37).  A  set 
percentage  of  academic  courses  and  trainers  are  evaluated  each  year,  as 
specified  in  SACR  60-4.  These  evaluations  focus  on  the  content  of  instruction 
and  the  presentation  of  materials  by  instructors.  They  also  serve  as  a  gauge 
of  program  implementation,  as  one  of  the  targets  of  evaluation  is  to  determine 
if  instructors  have  carried  out  lessons  as  prescribed.  Results  are 
disseminated  monthly  to  the  DO,  DOS,  and  WISH.  The  noted  discrepancies  and 
recommendations  for  corrective  actions  which  result  from  these  evaluations 
become  the  responsibility  of  DUS. 


Bomber  and  Tanker  Training  Program  Management  (D02B/K) 

D02B  and  D02K  are  the  training  program  managers  for  oombers  and  tankers, 
respectively.  They  work  with  HQ  SAC/DOTP  (Training  Programs)  to  build  each 
class,  taking  into  account  Coi.imano  requirements  and  LOTS  capabilities.  They 
also  act  as  CCTS  instructor-personnel  managers  and  work  to  maintain  required 
instructor-force  levels.  During  the  flightline  phase  of  training,  these 
managers  are  responsible  for  reviewing  and  acting  upon  Castle  Form  144s, 
Student  Action  Record,  submitted  via  the  Mission  Review  Panel  meetings.  Wing 
sched'ul i'ng  ( DDT )  is  "then  notified  of  any  events  in  wnich  additional  training 
is  required,  and  these  events  are  incorporated  into  subsequent  mission 
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development.  Castle  Form  144s  are  also  used  by  D02B/K  to  detect  trends  in  the 
flying  program.  For  example,  more  events  are  waived  during  the  winter  months, 
because  of  weather  conditions,  than  at  other  times  of  the  year.  The  presence 
of  trends  may  produce  changes  in  training  policy. 

D02B  and  D02K  are  also  responsible  for  reviewing  and  signing  TARs  prior 
to  solo  flights  and  for  obtaining  waivers  of  training  events.  When  TARs  are 
processed  at  the  conclusion  of  the  CCTS  program,  total  sorties  and  WST 
sessions  are  counted  for  tracking  purposes.  Students  are  issued  a  certificate 
of  flightline  course  completion  (DOT  FL  #7),  which  they  present  to  the 
329  CCTS,  and  a  Certificate  of  Training  (AF  Form  1256)  is  then  issued. 

D02B  and  D02K  prepare  CCTS  Graduate  Summaries  in  accordance  with  SACR 
51-52  and  SACR  51-135.  These  summaries  include  class  composition,  such  as 
total  crews  and  numbers  of  graduates  at  each  crew  position;  sorties  scheduled 
and  flown;  training  events  waived;  late  graduates;  and  S/E  class  and 
individual  results.  These  summaries  are  typically  sent  to  8  Air  Force  (AF) 
and  15  AF,  12  Air  Division  (AD),  Strategic  Air  Command  Headquarters  (HQ  SAC), 
and  Air  Training  Cointiand,  as  well  as  some  of  the  major  offices  within  the 
93  BMW. 


Instructional  Systems  Development  Division  (DOS) 

The  Instructional  Systems  Development  (ISD)  Division  (DOS)  is  the  focal 
point  for  development,  revision,  and  validation  of  the  aircrew  training 
curriculum.  This  division  has  a  role  in  the  internal  evaluation  process,  and 
it  has  exclusive  responsibility  for  the  external  evaluation  program.  SACR 
55-70  specifies  that  the  D05  is  also  responsiole  for  integrating  aircrew 
training  devices  into  the  training  curriculum  and  validating  this  training. 
DOSS  directs  and  conducts  FOT&E  for  aircrew  training  devices  such  as  the  B-52 
and  KC-135  WSTs. 

Validation  is  an  activity  that  is  integral  to  the  ISD  process  (AFM  50-2; 
AFP  50-58,  Vol.  IV).  A  formal  plan  is  usually  developed  for  validation 
efforts  at  the  93  BMW,  but  small  changes  in  the  curriculum  could  be 
incorporated  and  validated  without  a  formal  plan.  There  is  typically  a  range 
of  indices  from  which  to  assess  the  validity  of  a  particular  addition  or 
change  to  the  curriculum.  Most  often,  however,  assessments  consist  of  student 
and  instructor  questionnaires.  A  more  extensive  validation  effort  for  a 
course  in  the  navigator  curriculum  included  17  sources  of  data.  Information 
included  data  gathered  from  CAFB  Form  11  in  academic  ana  training  device 
portions  of  instruction;  interviews  with  students,  subject-matter  experts,  and 
instructors;  course  critiques;  courseware  and  task  audits;  and  external 
evaluations,  including  CCTS  Training  Effectiveness  Questionnaires  and  results 
from  needs-analysis  visits  to  gaining  units.  Direct  Knowledge  and  performance 
measures  were  not  among  the  data  used  to  validate  this  course,  however. 

D05  reports  that  pure  validation  efforts  are  difficult  to  implement  in 
practice,  as  student  flows  through  the  initial  qualification  training  program 
must  be  maintained.  As  a  result,  if  a  new  courseware  package  is  deficient  in 
some  respect,  the  instructor  must  take  responsibility  for  correcting  the 
deficiencies  on  the  spot  and  provide  any  instruction  required  for  the  student 
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to  pass  the  course.  Accordingly,  such  deficiencies  may  not  be  detected  by  an 
analysis  of  examination  scores.  Under  these  conditions,  instructor 
questionnaires  assume  increased  importance  in  tne  validation  effort.  More 
generally,  examination  scores  and  proficiency  ratings  from  aircrew  training 
devices  are  not  used  at  all  in  validation  efforts,  just  as  they  are  not  used 
in  the  ongoing  evaluation  of  training  system  effectiveness. 

According  to  ISD  procedures  (AFil  50-i^),  evaluation  is  conceptualized  as 
commencing  after  programs  have  been  validated.  It  consists  of  two  types: 
internal  evaluation  and  external  evaluation  (AFP  5U-5B,  Vol.  V).  In  its  iiiost 
general  sense,  internal  evaluation  is  conducted  within  the  93  BMW.  External 
evaluation  is  conducted  in  the  field,  and  it  is  an  assessment  of  the  extent  to 
which  the  training  at  CCTS  successfully  prepares  graduates  to  perform  their 
flying  duties  at  B-52  and  KC-135  operational  units. 

There  are  several  types  of  internal  evaluations,  soiiie  of  which  have 
already  been  discussed,  and  D05  participates  in  most  of  them.  These  types  of 
internal  evaluations  are  separate  from  the  SACR  bO-4  performance  evaluations 
conducted  by  DOV,  although  the  latter  can  also  be  considered  as  internal 
evaluations,  since  they  are  conducted  within  the  93  BMW. 

1.  In  the  student  critique  process,  DOS  is  the  first  and  last  office  to 
receive  completed  student  critique  forms  from  academics  and  the  flightline. 

By  being  the  first  office  to  receive  these  forms,  DOS  can  begin  coordinating 
changes  to  the  curriculum  almost  immediately. 

2.  As  discussed  earlier,  DOV  conducts  educational  evaluation  activities 
which  are  documented  on  Castle  Form  11.  DOS  is  responsible  for  coordinating 
changes  in  the  program  which  result  from  these  evaluations. 

3.  Any  agency  or  individual  may  suggest  modifications  to  the  training 
curriculum  at  any  time,  and  this  is  done  by  completing  Castle  Form  211, 
Recoiiriendation  for  Training  Improvement  (see  Appendix,  p.  38).  DOS  provides 
an  initial  response  to  each  suggestion  that  is  submitted.  The  approval  of  the 
CRB  is  required  to  institute  major  changes  to  the  curriculum,  and  this  body 
also  arbitrates  any  differences  of  viewpoint  regarding  potential  changes. 

4.  DOS  is  responsible  for  reviewing  the  currency  of  courseware  at  least 
annually  or  when  changes  are  necessary  for  such  reasons  as  modifications  to 
the  aircraft.  These  reviews  are  recorded  on  Castle  Form  371,  Courseware 
Review/Revision  Checklist.  DOS  reports  that,  in  practice,  external  changes 
can  force  courseware  reviews  with  such  frequency  that  nearly  every  course  is 
"reviewed"  at  least  once  annually  from  this  process  alone.  In  addition,  all 
courseware  must  be  inventoried  annually,  and  tnis  involves  updating  Castle 
Form  42,  KC-13S  CCTS  Student  Publications,  and  Castle  Form  42a,  B-52  Student 
Publ ications.  The  inventory  of  courseware  is  actually  a  type  of  tracking 
activity  and  is  not  an  evaluation. 

5.  An  additional  type  of  internal  evaluation  is  a  critique  of  base 
facilities  which  is  completed  by  students  and  forwarded  to  the  Director  of 
Academics.  The  results  of  these  critiques  are  sutih.iarized,  and  they  are  sent 
to  the  Base  Commander  for  review.  DOS  has  no  role  in  this  process,  however. 


DOS  has  exclusive  responsibil ity  for  external  evaluation.  Three 
methods  of  external  evaluation  are  listed  in  AFM  50-2  and  AFP  50-5?,  Vol.  V: 
questionnaires,  field  visits,  and  job  performance  evaluations.  Only  the  first 
two  types  of  evaluation  are  conducted  as  part  of  the  y3  BMW  external  evaluation 
program.  The  completion  of  CCTS  Training  Effectiveness  Questionnaires  is 
required  by  SACR  51-52  and  SACK  51-135.  These  questionnaires  are  completed  by 
graduates  within  30  days  after  having  attained  mission-ready  status  at  their 
gaining  units.  An  external  evaluation  questionnaire  is  also  completed  by 
gaining-unit  squadron  commanders  or  a  designated  representative.  The 
designated  representative  is  most  often  the  Training  Flight  instructor  who 
conducted  the  local  unit  upgrade  of  tne  CCTS  graduate.  Separate 
questionnaires  have  been  designed  for  B-52  and  KC-135  aircrews.  With  few 
exceptions,  most  of  the  questions  apply  to  all  crew  positions  for  a  given 
aircraft.  Content  items  are  rated  on  a  5-point  scale.  Generally,  the  items 
apply  to  phases  of  flight  such  as  mission  planning,  preflight,  and  takeoff. 
Other  items  concern  knowledge  and  skill  areas  such  as  crew  coordination, 
emergency  procedures,  and  air  refueling.  Items  which  are  rated  low  require  a 
written  comment,  presumably  to  determine  the  specific  skills  in  which 
graduates  are  deficient.  Background  information  is  also  obtained  to 
determine,  for  example,  how  much  flying  graduates  have  actually  accomplished 
since  leaving  Castle  AFB.  All  information  is  entered  on  a  computer  form  v/hich 
is  returned  to  the  93  BMW.  The  results  are  analyzed  and  presented  for  each 
class  in  the  minutes  of  the  CRB  and  TRP.  Final  results  are  presented 
separately  for  graduates  and  squadron  commanders.  Return  rates  and  average 
ratings,  collapsed  across  crew  positions,  are  listed.  Specific  problem  areas 
within  a  crew  position  are  also  identified  if  they  emerge. 

Periodic  needs-analysis  visits  to  operational  units  are  conducted  as  a 
second  type  of  external  evaluation.  These  evaluations  are  performed  in 
accordance  with  AFR  50-38  and  SACR  55-70.  CCTS  graduates  and  personnel  from 
operational  units  are  interviewed  to  obtain  specific  information  about  the 
effectiveness  of  training  at  Castle  AFB.  In  particular,  evaluators  attempt  to 
ascertain  what  skills  should  be  taught  that  are  not  currently  incorporated 
into  the  program  at  the  93  BMW.  Written  reports  from  these  visits  are 
completed  upon  return,  and  they  include  a  synopsis  which  identifies  areas  of 
concern  for  each  crew  position.  The  body  of  each  report  lists  the  number  of 
graduates  at  each  crew  position  who  were  interviewed,  wing  personnel 
interviewed,  and  specific  areas  of  concern.  Reports  of  results  are 
distributed  to  major  offices  within  the  93  BMW,  and  to  HQ  SAC,  15  AF,  and  12 
AD.  All  reports  are  maintained  in  a  file  at  tne  D05. 

The  12  AD  and  Command  CRBs  are  additional  forums  for  external  evaluation, 
but  they  are  informal  means  of  evaluation.  Changes  in  job  requirements  may 
also  be  included  in  discussions  at  these  meetings. 


Curriculum  Review  Groups,  the  Curriculum  Review  Board,  and  the  Training 
Review  Fanel 

The  CRGs,  CRB,  and  TRP  are  the  formally  established  bodies  at  tne  93  BMW 
which  review  the  status  of  the  entire  aircrew  training  curriculum  and  make 
decisions  affecting  the  programs.  The  CCTS  review  process  occurs  in  a 
hierarchical  fashion.  D02B  and  D02K  chair  the  CRGs.  These  groups  are 
considered  as  the  working  groups  of  the  CRB.  The  Deputy  Comiaander  for 
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Operations  chairs  the  CRB.  The  Winy  Coi.ii.iander  chairs  tne  TRP  which  is  the 
highest  level  of  review  of  the  CCTS  prograi.i  within  the  93  BMW. 


As  a  iiinirninn,  tlie  items  of  information  tracked  and  published  by  these 
bodies,  in  addition  to  discussion  items  from  the  CRGs  and  CRB,  are:  (a)  number 
of  classes  graduated;  (b)  external  evaluation  results;  (c)  flying  hours  and 
sorties  requested  and  flown  for  each  month;  (d)  student  loads  per  class;  and 
(e)  a  3-month  recap  of  5/E  results,  with  areas  of  interest  and  weaknesses 
identified  by  the  trends  analysis  progi^am.  Class  Summaries,  described  above, 
are  attached  to  the  TRP  report. 


V.  SYNOPSIS  OF  INFORMATION  SYSTEM 

The  essential  data  items  comprising  the  information  system  will  now  be 
categorized  in  order  to  provide  a  summary  statement  of  the  system  and  make  it 
more  amenable  to  analysis  in  the  Discussion.  In  accordance  with  tne  stated 
purpose  of  the  present  effort,  the  emphasis  is  on  the  evaluation  function. 
Table  1  presents  three  categories  of  information:  student  evaluation,  program 
evaluation,  and  management  of  student  instruction.  The  main  data  items  in 
each  category  are  enumerated,  while  items  not  serving  critical  functions  in 
these  processes  are  deleted  from  the  presentation. 

Student  Evaluation.  There  are  ai.iple  provisions  for  evaluating  student 
learning  and  performance  ttiroughout  the  training  program  at  tne  93  BMW. 
Examination  scores  and  the  results  of  sessions  in  aircrew  training  devices  are 
recorded  in  the  329  CCTS.  In  the  flightline  phase  of  training,  proficiency 
ratings  for  each  WST  and  aircraft  event  are  recorded  on  progress  sheets,  and 
flying  training  events  are  logged  on  the  TAR.  The  point  at  which  proficiency 
is  attained  for  a  flying  event  is  also  indicated  on  the  TAR.  The  SACR  60-4 
evaluation  is  conducted  after  the  flying  phase  of  training  is  completed. 

There  are  also  provisions  for  recording  qualitative  information  with  these 
flying  evaluation  procedures.  Critique  items  are  noted  on  the  progress  sheets 
and  the  SACR  60-4  evaluations. 

Program  Evaluation:  Process  and  Outcome.  Table  1  lists  four  types  of 
process  evaluations  and  three  types  of  outcome  evaluations  used  for  program 
evaluation.  Process  evaluations  include  student  critiques  of  the  academic  and 
flightline  phases  of  instruction,  educational  evaluations  and  evaluations  of 
training  devices  conducted  by  DOV,  and  recommendations  for  training 
improvement.  Outcome  measures  used  for  program  evaluation  are  SACR  60-4 
evaluations  and  the  associated  trends  analysis  results,  the  results  from 
training  effectiveness  questionnaires,  and  results  of  needs-arialysis  visits  to 
gaining  units.  The  SACR  60-4  evaluations  and  the  trends  analysis  results  are 
the  only  data  wnich  come  from  direct  performance  evaluations.  Evaluations 
receiving  the  greatest  visibility  within  the  Wing  are  student  critiques  of 
training,  SACR  60-4  evaluations  and  trends  analysis  results,  and  training 
effectiveness  questionnaires.  It  is  important  to  emphasize  that  examination 
scores  from  academics  and  the  performance  data  collected  during  instructional 
sessions  in  aircrew  training  devices  and  the  aircraft  are  not  used  in  formal, 
systematic  program  evaluation. 

Management  of  Student  Instruction.  The  completion  of  blocks  of  acadetiiic 
instruction  and  sessions  in  aircrew  training  devices  are  tracked  on  the 
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Table  1:  Summary  of  Primary  Items  of  Information 
Student  evaluation 

-  Examination  scores  and  ATD  results  in  academics 

-  Progress  records  for  WSTs  and  aircraft  sorties/ 

Training  Accomplishment  Reports 

-  SACR  60-4  individual  evaluations  (AF  Form  8) 

Program  evaluation:  process  and  outcome 

-  Student  critiques  of  academic  and  flightline  instruction 

-  S/E  educational  evaluations  and  evaluations  of  training  devices 

-  Recommendations  for  training  improvement 

-  SACR  60-4  evaluations  (Class  Summary)  and  trends  analysis  results 

-  Training  effectiveness  questionnaires:  graduates  and  squadron 
commanders 

-  Needs-analysis  visits  to  gaining  units 
Management  of  student  instruction 

-  Criterion  test  record:  examinations  and  ATD  sessions 

-  Training  Accomplishment  Reports/Progress  Records 


criterion  test  record  in  the  329  CCTS.  The  TAR  and  progress  records  are  used 
to  track  event  accomplishments  and  the  attainment  of  proficiency  for  WST  and 
flying  events  during  the  aircraft  phase  of  instruction,  and  they  also  provide 
information  used  to  schedule  flying  events.  Flight  commanders  and  DQ2B/K  sign 
all  TARs  at  the  conclusion  of  the  training  program,  to  ensure  that  all  flying 
instruction  has  been  received  or  event  waivers  have  been  obtained. 


VI.  DISCUSSION 

It  will  be  recalled  from  the  Introduction  that  judgments  of  program 
effectiveness  are  dependent  primarily  on  outcome  measures  (Patton,  1986, 
p.  345;  Scriven,  1982,  p.  48).  Why  or  how  a  prograiii  is  effective  or 
ineffective  is  determined  through  an  evaluation  of  program  processes  and 
operations  (Judd,  1987).  Process  evaluations  are  also  useful  for  improving 
program  effectiveness  and  efficiency.  Ideally,  both  types  of  evaluation 
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should  be  combined  in  program  evaluations  (Cronbach,  1982;  Judd,  1987; 

Scriven,  1982,  p.  121).  Properly  used,  the  tvw  data  sets  are  interactive.  A 
full  interpretation  of  outcomes  or  program  effectiveness  is  dependent  upon  the 
results  of  process  evaluations,  and  the  improvement  of  a  program,  engendered 
primarily  through  process  evaluations,  must  be  gauged  relative  to  subsequent 
outcomes. 

Fairly  comprehensive  sets  of  student  learning  and  performance  data  are 
currently  collected  throughout  all  phases  of  training  at  the  93  BMW  (see  Table 
1:  Student  Evaluation).  Although  all  of  these  data  are  used  to  evaluate 
student  learning  and  performance  and  to  manage  the  program  of  instruction, 

SACR  60-4  checkride  results  are  the  only  outcome  measures  used  in  formal, 
internal  evaluations  of  program  effectiveness.  Academic  test  scores  and 
proficiency  measures/event  accomplishments  from  aircrew  training  devices  and 
the  aircraft  are  not  aggregated  and  analyzed  to  provide  estimates  of  training 
effectiveness  associated  with  these  phases  of  instruction.  This  replicates 
the  finding  of  Bruce  et  al.  (in  preparation),  where  it  was  determined  that  the 
main  function  of  knowledge  and  performance  evaluations  at  SAC  operational 
units  is  to  assess  the  competence  of  aircrews,  not  necessarily  the 
effectiveness  and  efficiency  of  training.  Training  effectiveness 
questionnaires  and  needs-analysis  visits  to  gaining  units  are  additional 
outcome  evaluations  used  to  assess  the  effectiveness  of  the  CCTS  program,  but 
they  are  used  to  evaluate  the  program  external ly--not  internally. 

Sole  reliance  on  using  SACR  60-4  performance  assessments  and  the 
associated  trends  analysis  results  for  evaluations  of  internal  program 
effectiveness  can  be  likened  to  a  "black-box"  approach  to  evaluation.  The 
focus  is  on  the  terminal  outcome  measures  alone,  since  these  measures  occur  at 
the  conclusion  of  a  fairly  lengtny  process  of  training  which  contains  multiple 
phases  and  components.  The  SACR  60-4  outcome  measures  are  not  referenced  to 
any  process  evaluation  results  or  descriptions  of  current  training  practices, 
and  other  more  localized  outcome  measures  which  correspond  to  the  individual 
phases  or  components  of  instruction  are  not  considered  at  all  oy  this  method 
of  program  evaluation.  Hence,  the  nature  of  the  training  program  remains  a 
"black  box."  One  may  know  that  training  is  effective  or  ineffective,  but  not 
what  type  of  training  has  been  proved  effective  or  why  it  is  effective. 
Additional  limitations  of  black-box  approaches  to  evaluation  have  been 
discussed  by  other  evaluators  (e.g.,  Patton,  1986,  pp.  122-149;  Scriven,  1982, 

p.  18). 

There  are  other  consequences  of  considering  only  checkride  and  associated 
trends  analysis  results  when  evaluating  internal  program  effectiveness. 

First,  SACR  60-4  evaluation  results  must  be  considered  as  only  general  outcome 
measures  which  represent  the  combined  effects  of  exposure  to  the  entire 
training  regimen.  From  these  results,  one  cannot  make  conclusions  about  the 
effectiveness  of  individual  components  of  the  training  system  or  the  transfer 
of  training  between  components.  It  would,  therefore,  be  tenuous  to  assume 
that  the  entire  training  system  was  effective  based  on  these  results  alone. 

It  is  possible  that  one  phase  or  component  of  instruction  could  compensate  for 
an  earlier,  inadequate  phase  of  instruction,  although  the  net  effect  of 
instruction  in  these  components  may  be  to  produce  performance  sufficient  to 
pass  a  checkride.  For  example,  one  could  not  assume  that  current  versions  of 
WST  training  were  effective  simply  because  students  successfully  passed 
checkrides.  The  effectiveness  of  WST  training  would  require  a  detailed 
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evaluation  in  its  own  right,  including  an  analysis  of  performance  measures 
collected  during  simulator  and  aircraft  phases  of  instruction.  Indeed, 
determinations  of  the  effectiveness  of  aircrew  training  devices  at  the  93  BMW 
have  relied  on  such  independent  evaluations  (Bruce  &  Keyes,  in  preparation; 
Gray,  1979;  Nullmeyer  &  Laughery,  1980),  but  they  are  conducted  very 
infrequently,  typically  as  part  of  Operational  Test  and  Evaluation.  The 
design,  operation,  and  improvement  of  a  fully  effective  training  system 
requires,  however,  the  continuous,  formal  evaluation  of  all  components  and 
phases  of  instruction,  separately  and  in  combination,  particularly  since 
program  operations  are  likely  to  change  over  time. 

Another  consequence  of  reliance  on  SACR  60-4  checkride  and  trends 
analysis  results  is  that  these  evaluations  assess  only  a  sample  of  the  total 
knowledge  and  skills  required  of  graduates.  If  these  assessments  included  the 
entire  spectrum  of  knowledge  and  skills  in  the  formal  school  program,  they 
would  be  forbiddingly  expensive  in  time  and  other  program  resources. 
Accordingly,  one  must  conduct  other  supplementary  evaluations  to  determine  if 
the  system  effectively  trains  knowledge  and  skills  not  contained  in  the 
typical  checkride.  This  issue  is  beginning  to  be  discussed  more  fully  with 
the  advent  of  contracted  aircrew  training,  as  it  is  particularly  relevant  in  a 
contractual  sense  (R.T.  Nullmeyer,  personal  communication,  1987).  For 
example,  if  a  contractor  is  responsible  for  training  a  specified  set  of  skills 
or  ensuring  that  students  meet  set  performance  objectives,  tnere  must  be  other 
means  of  assuring  that  these  obligations  have  been  met,  if  only  a  portion  of 
the  objectives  or  skills  is  sampled  by  the  checkride.  One  way  of  doing  this 
is  to  use  the  appropriate  outcome  data  associated  with  tne  individual 
components  of  instruction. 

The  trends  analysis  program  is  considered  a  principal  means  of  detecting 
and  correcting  deficiencies  in  the  training  program.  Realistically,  since 
trends  analysis  results  are  based  solely  on  outcome  measures  that  are 
collected  at  the  conclusion  of  training,  they  can  be  used  only  to  detect  that 
a  problem  may  potentially  exist  with  the  training  system.  By  themselves, 
these  data  provide  little  information  for  effective  action.  Other  kinds  of 
evaluative  data  are  required  to  determine  where  in  the  system  a  training 
problem  may  reside,  the  precise  nature  of  the  problem,  and  how  it  might  be 
corrected.  In  a  systematic  approach  to  problem  diagnosis  and  correction,  one 
must  first  validate  that  a  problem  exists  with  the  training  system,  for  the 
identified  problem  could  be  an  effect  specific  to  the  SACR  60-4  evaluation 
process  itself  or  some  other  factor.  Next,  a  determination  of  where  in  the 
training  system  the  problem  may  reside  would  require  local  outcome  measures  or 
other  data  associated  with  the  various  components  in  academics,  simulators, 
and  the  aircraft.  Once  the  locus  had  been  established,  it  would  then  be 
necessary  to  have  information  about  the  processes  of  instruction  associated 
with  the  defective  componentls)  in  order  to  determine  the  precise  nature  of 
the  training  problem.  If  data  were  unavailable,  independent  observations  of 
training  would  have  to  oe  made  to  suggest  possible  areas  of  difficulty.  The 
process  information  would,  in  turn,  be  useful  in  devising  alternative 
solutions  to  correct  the  training  problem.  Once  changes  to  the  system  were 
implemented,  localized  output  measures  associated  with  the  previously 
defective  component  could  then  oe  monitored.  These  measures  could  be  compared 
with  subsequent  trends  analysis  results  to  confirm  that  the  training  problem 
had  been  resolved.  One  could  hypothesize  that  routine  collection  and  use  of 
localized  measures  of  component  effectiveness  might  reduce  the  number  of 
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trends  detected  by  the  trends  analysis  program,  although  this  is  subject  to  an 
empirical  test.  Such  a  use  of  localized  measures  would  be  considered 
preventive,  since  problems  would  not  need  to  await  detection  by  the  trends 
analysis  program. 

A  systematic  diagnostic  and  change  process  such  as  tnat  described  above 
does  not  typically  occur,  however.  There  is  considerable  variation  in  the 
action  applied  to  trends,  but  it  is  not  uncommon  for  a  short-circuiting  of  the 
involved  trouoleshooting  and  change  process  to  occur.  For  example,  instructors 
nay  be  requested  to  "re-emphasize"  certain  aspects  of  training,  but  this  is  no 
real  solution  for  correcting  a  potential  deficiency  in  the  training  system. 
Further,  the  effects  of  such  adriionishments  can  be  short-lived.  Significant 
deficiencies  in  a  training  program  require  a  more  systematic  approach  for 
effective  and  durable  resolution. 

The  remaining  sets  of  data  used  for  internal  evaluation--student 
critiques,  S/E  academic  evaluations  and  evaluations  of  training  devices,  and 
recomnendations  for  training  improvement--relate  to  the  processes  of 
instruction.  As  such,  they  assess  the  manner  in  which  instruction  is 
delivered,  whether  instructors  are  helpful,  if  instructional  materials  and 
test  items  are  understandable,  and  other  targets  of  evaluation.  All  phases 
and  components  of  instruction  are  evaluated--at  least  partial ly--by  these 
methods.  Importantly,  these  evaluations  also  reflect  a  number  of  viewpoints, 
such  as  those  of  students,  experienced  evaluators,  and  instructors. 

In  practice,  the  process  evaluations  (especially  student  critiques) 
are  often  interpreted  as  indicators  of  training  effectiveness  at  the  93  BMW. 

It  is  important  to  re-emphasize,  however,  that  determinations  of  effectiveness 
relate  to  the  observed  outcomes  of  instruction  (see  Patton,  1986,  p.  345; 
Scriven,  1982,  p.  48).  Accordingly,  measures  of  student  learning  and 
performance  are  essential  in  making  these  judgments.  Even  if  student  critiques 
are  viewed  by  tlie  students  themselves  as  devices  to  record  impressions  of 
training  effectiveness  (which  they  often  are),  one  can  seriously  question  the 
capability  of  students  to  render  such  judgments,  particularly  during  initial 
qualification  training.  The  legitimate  use  of  student  critiques  as  measures 
of  program  effectiveness  would  assume  a  valid,  internalized  set  of  knowledge 
and  performance  standards  on  the  part  of  students,  which  is  very  tenuous 
indeed.  More  likely,  experienced  external  evaluators  are  required  to  make 
valid  assessments  of  whether  students  are  performing  in  accordance  with 
established  standards,  and  whether  instruction  is  effectively  producing  those 
outcomes  (M.R.  Rockway,  personal  communication,  1986).  These  comments  are  not 
intended  to  denigrate  the  importance  of  student  evaluations  of  instruction; 
rather,  they  are  intended  to  clarify  the  nature  of  the  judgments  which  can  be 
derived  from  such  critiques.  Many  important  results  of  instruction,  including 
unintended  ones,  can  be  detected  from  these  types  of  evaluations.  This 
information  is,  in  turn,  crucial  for  designing  effective  and  efficient 
training  systems. 

Another  common  practice  at  the  93  BMW  is  that  the  results  of  process 
evaluations  are  acted  upon  independently  of  outcome  results  associated  with 
the  same  components  or  phases  of  instruction.  This  is  largely  because  the 
outcome  measures  are  used  for  student  evaluation  and  instructional  management, 
but  it  is  also  because  some  of  the  process  evaluations  are  administered 
separately  by  different  organizations  (e.g.,  DOV)  within  the  93  BMW.  The 
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consequence  of  these  parallel  evaluation  functions  is  that  the  program  may  be 
modified  solely  on  the  basis  of  process  evaluations  such  as  student 
critiques.  Comparisons  between  process  and  outcome  ineasures  associated  v/ith 
the  same  components  of  instruction  can  be  used  in  some  cases  to  verify  tnat  a 
component  is  functioning  problematically.  When  using  process  evaluations  for 
the  subsequent  improvement  of  instruction,  individuals  who  use  sucii 
information  may  take  into  account  the  likely  outcomes  of  resulting  alterations 
to  instruction,  but  it  is  important  that  such  changes  also  be  directly  related 
to  observed  outcomes.  Improved  effectiveness  or  greater  efficiencies  could 
then  be  confirmed,  or  changes  which  produce  subsequent  decreiiients  in 
effectiveness  or  efficiency  could  be  documented  for  future  reference. 

The  formal  school  is  part  of  a  larger  continuum  of  training  for  B-b2  and 
KC-135  which  includes  mission  qualification  and  continuation  training. 

External  evaluation  is  one  mechanism  by  which  impacts  on  the  larger  training 
system  can  be  assessed.  Adjustments  or  additions  to  the  formal  school  program 
can  also  be  made  on  the  basis  of  these  evaluations.  External  evaluation  is  an 
important  part  of  the  overall  program  evaluation  at  the  9J  BilW,  as  training 
may  be  judged  effective  internally,  yet  fail  to  meet  the  expectations  and 
actual  needs  of  the  operational  units.  The  consequences  of  insufficient 
training  at  the  formal  sciiool  are  serious,  as  training  resources  are 
comparatively  scarce  at  the  gaining  units  (Bruce  et  al.,  in  preparation). 

Both  training  effectiveness  questionnaires  and  neeas-analys is  visits  are 
essential  components  of  external  evaluation.  They  focus  on  different,  yet 
complementary,  aspects  of  external  evaluation.  Training  effectiveness 
questionnaires  attempt  to  assess  how  well  graduates  can  actually  perform  at 
the  gaining  unit  those  tasks  which  are  currently  trained  at  the  formal 
school.  Needs-analysis  visits,  by  contrast,  are  geared  toward  ascertaining 
other  program  needs  which  are  not  incorporated  into  the  existing  training 
program.  Needs-analysis  information  is  vital,  as  there  may  ue  changes  in  the 
nature  of  missions  at  operational  units  which  require  either  different  skills 
or  extensions  of  skills  already  trained  in  the  CCTS  program.  This  is 
especially  pertinent  considering  the  increasing  conventional  role  of  the  B-b2 
(Bruce  et  al.,  in  preparation).  The  full  range  of  needs  at  operational  units 
must  also  be  assessed,  as  there  are  variations  in  mission  types  from  unit  to 
unit.  Accordingly,  needs-analysis  visits  must  incorporate  clear  provisions 
for  frequent  sampling  across  a  wide  range  of  operational  units. 

The  actual  allocation  of  additional  training  requirements  to  the  formal 
school  is  a  complex  process.  The  entire  continuutn  of  training  for  B-b2  and 
KC-13b  should  be  considered  in  making  these  decisions,  and  they  should  be 
based,  in  part,  on  the  results  of  needs-analysis  visits  and  current  CCTS 
training  requirements.  Resources  which  are  available  for  use  in  training  at 
selected  points  throughout  the  entire  system  must  also  be  considered.  In  this 
regard,  the  formal  school  nas  a  greater  concentration  of  training  resources 
than  gaining  units.  Additional  factors  to  be  considered  at  operational  units 
may  include  current  training  requirements,  alert  requirements,  and  other 
activities  which  directly  compete  with  the  accomplishment  of  training.  The 
latter  factor  includes  higher  headquarters  directives  and  special  missions 
(Bruce  et  al.,  in  preparation). 

The  needs-analysis  visits  conducted  by  the  93  BilW  occur  infrequently. 

In  the  absence  of  more  frequent  information  from  needs-analysis  visits  or 
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information  provided  directly  by  operational  units,  the  12  AD  and  Command  CRBs 
and  Wing  Commander  conferences  furnish  other  opportunities  for  cross-checks  in 
this  important  area  of  evaluation.  Needs  identified  in  these  forums  could  be 
systematically  combined  and  documented  with  those  of  needs-analysis  visits,  to 
provide  a  more  comprehensive,  running  account  of  overall  program  needs. 

Results  of  needs-analysis  visits  should  also  receive  visibility  comparable  to 
those  of  training  effectiveness  questionnaires  which  are  published  in  the  CRB 
and  TRP  minutes  at  the  93  BMW.  Such  a  practice  would  permit  a  fuller 
interpretation  and  demonstration  of  the  external  value  of  trie  CCTS  program. 

Training  effectiveness  questionnaires  are  completed  by  graduates  and 
gaining-unit  squadron  commanders.  Since  graduates  have  completed  one  or  more 
flying  missions  at  gaining  units  prior  to  filling  out  these  questionnaires, 
they  have  a  more  valid  framework  for  determining  if  training  at  the  93  BMW 
adequately  prepared  them  to  perform  their  flying  duties.  Questionnaires 
completed  by  squadron  commanders  can  be  particularly  important  sources  of 
information,  since  they  are  completed  by  operational  personnel  external  to  the 
93  BMW.  Their  value  is  dependent,  however,  on  the  use  of  additional  data  by 
squadron  commanders.  These  data  could  consist  of  proficiency  ratings  for 
various  flying  events,  critique  items,  or  other  evaluative  data  collected  on 
incoming  graduates  of  the  CCTS  program.  The  use  of  these  data  would  ensure 
more  valid  ratings  on  the  questionnaire  itself.  Ideally,  external  evaluators 
should  be  instructed  to  use  all  available  information  in  completing  these 
questionnaires,  and  they  should  also  identify  what  particular  items  of 
information  were  actually  used.  Such  a  procedure  would  assist  in  the 
interpretation  of  external  evaluation  results. 

The  discussion  thus  far  has  not  been  intended  to  suggest  that  shortfalls 
in  current  evaluation  practice  at  the  93  BMW,  such  as  the  reliance  on  SACR 
60-4  evaluations  in  determinations  of  program  effectiveness,  result  from  an 
inadequate  conceptualization  of  evaluation.  Evaluation  practice  and  changes 
to  it  typically  result  from  practical  considerations  such  as  the  requirements 
for  information.  The  reliance  on  SACR  60-4  evaluation  results  for  assessments 
of  program  effectiveness  has  been,  until  recently,  a  defensible  practice. 
Despite  their  limitations,  these  evaluations  are  good,  general  measures  of 
program  effectiveness.  Deficiencies  in  flying  can  be  detected  from  them,  and 
they  are  economical  relative  to  more  precise  component  and  system  evaluations 
which  require  more  evaluation  resources.  The  consideration  of  evaluation 
resources  suggests  that  evaluation  practice  be  viewed  as  a  trade-off  between 
information  and  evaluation  requirements  and  tiie  available  resources  to  conduct 
evaluation  (Cronbach,  1982).  Accordingly,  given  past  evaluation  requirements 
and  available  resources,  the  practice  of  using  checkride  results  alone  to 
evaluate  training  system  effectiveness  is  appropriate,  and  it  nas  worked  for 
many  years.  But  the  situation  has  changed  significantly.  Recent  requirements 
for  evaluations  of  aircrew  training  devices,  the  advent  of  contracted  aircrew 
training,  and  the  likely  requirement  to  more  effectively  justify  training 
resources  such  as  flying  hours  renders  this  approach  to  evaluation 
insufficient.  Checkride  results  alone  do  not  provide  the  needed  information. 
Evaluations  which  assess  training  components,  their  interactions,  and  the 
training  system  as  a  whole  are,  therefore,  likely  to  become  the  standard 
(e.g..  Aeronautical  Systems  Division,  1986;  Fishburne  et  al.,  1987,  pp.  29-40; 
Spears,  1986).  This  will  require,  however,  an  investment  in  more  evaluation 
resources.  Evaluation  must  tnen  demonstrate  a  responsiveness  to  these 
challenges;  that  is,  it  must  be  worth  the  investment. 
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Perhaps  the  principal  point  to  be  raised  with  respect  to  evaluation  at 
the  93  BMW  is  that  the  Wing  currently  collects  most  of  the  primary  information 
from  which  more  detailed  component  and  system  evaluations  can  be  conducted. 

As  we  have  seen,  examination  scores  in  academics,  and  proficiency  ratings  and 
event  accomplishments  in  aircrew  training  devices  and  the  aircraft,  are 
routinely  recorded,  although  they  are  used  for  different  purposes  (i.e., 
student  evaluation  and  management  of  instruction).  These  treasures  are,  in 
fact,  the  primary  measures  upon  wnich  previous  effectiveness  evaluations  of 
aircrew  training  devices  at  the  93  BMW  have  relied  (e.g.,  Bruce  &  Keyes,  in 
preparation;  Wullmeyer  &  Laughery,  1980).  These  evaluations  not  only 
demonstrated  the  effectiveness  of  expensive  aircrew  training  devices,  but  they 
also  suggested  how  the  devices  should  be  used.  These  types  of  evaluations 
should  be  conducted  routinely,  as  information  of  such  importance  should  not  oe 
dependent  upon  infrequent  FUT&Es.  What  we  are  trying  to  affect,  then,  is 
information  use,  as  the  Wing  has  already  invested  resources  in  collecting  the 
information.  Accordingly,  our  previous  discussion  can  be  considered,  in  part, 
a  justification  for  using  all  tne  available  evaluative  infonnation,  and  using 
it  in  a  certain  manner--to  evaluate  student  progress  and  the  training  system. 


There  are  two  important  areas  which  need  to  be  addressed  in  order  to 
implement  the  prescribed  use  of  training  information.  The  current  manual 
record  keeping  system  at  the  93  BMW  would  make  routine  processing  and  analysis 
of  examination  scores,  proficiency  ratings,  and  event  accomplishments 
extremely  labor-intensive.  The  problem  of  manual  record  keeping  systems  was 
also  noted  in  the  assessment  of  current  C-130  training  (Fishburne  et  al., 

1987,  p.  81).  This  situation  is  particularly  acute,  since  the  annual 
throughput  rate  for  initial  qualification  and  pilot/navigator-upgrade  training 
at  the  93  BMW  is  nearly  3,000  students.  Computer  support  is,  therefore, 
essential.  The  requirements  for  computer  assistance  would  increase  further  i^" 
all  training  information  and  evaluation  results  were  included  in  a  central, 
computerized  database.  Tnis  automation  would  permit,  however,  the  coniinuous 
tracking  and  assessment  of  the  effectiveness  of  instructional  components  and 
the  training  system  as  a  whole.  Automation  would  also  permit  more  rapid 
detection,  diagnosis,  and  follow-up  of  training  system  problems  and  furnish  a 
moans  of  permanent  documentation.  More  generally,  it  would  be  a  step  toward 
unifying  the  entire  information  and  evaluation  system.  This  is  an  important 
benefit,  as  it  was  mentioned  previously  that  many  of  the  evaluations  are 
currently  conducted  and  used  in  a  parallel  fashion. 

The  second  area  which  needs  to  be  addressed  is  that  evaluation  functions 
are  now  seriously  fragmented  throughout  the  93  BMW.  Separate  evaluations  are 
conducted  by  the  329  CCTS,  93  AREFS,  328  BMS,  UOV,  and  DOS.  This  practically 
ensures  that  many  appropriate  and  informative  comparisons  of  evaluation 
results  collected  by  these  different  offices  will  not  be  made,  and  that 
responsibility  for  action  on  the  basis  of  evaluation  results  will  be  too 
diffuse.  Utilization  of  a  central  database  for  program  evaluation  would  also 
seem  to  require,  then,  a  centralized  office  for  processing  and  analyzing  all 
evaluation  results,  and  coordinating  and  managing  the  use  of  evaluative 
information  within  the  training  system. 

There  is  an  additional  ingredient  wnich  is  necessary  for  the  effective 
functioning  of  the  evaluation  system:  the  development  of  a  comprehensive, 
integrated  evaluation  plan.  Such  a  plan  is  necessary,  given  the  complexities 
of  an  aircrew  training  system,  current  training  and  evaluation  requirements  at 
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a  variety  of  levels,  and  the  diverse  informational  needs  of  curriculum 
developers,  evaluators,  training  managers,  and  senior  Wing  administrators.  In 
essence,  integrated  training  systems  require  integrated  evaluation  plans  vjhich 
serve  as  effective  guides  for  coordinated  information  collection,  processing, 
and  use  (Nullraeyer,  McGann,  &  Rooney,  1986).  The  development  of  this  plan  is 
to  be  an  important  part  of  the  upcoming  CCTS  Modernization  effort. 
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GLOSSARY  OF  ACRONYMS 


AO 

AF 

AFI1 

AFORIIS 

AFP 

AFR 

ARLFS 

ATC 

ms 

BMW 

CAFBR 

CCTS 

CPT 

CRB 

CRG 


DG 

DO 

DOT 

DOTB 

DOTF 

UOTK 

UOV 

OOVB 

DOVK 

002B 

002K 

005 

005S 


FJ  Tilt 

GAO 

Ho  SAC 

ISO 

MAR 

OT&E 


PR 

SAC 

S/l 


Air  Division 

Air  Force 

Air  Force  Manual 

Air  Force  Operational  Resource  ildnagei.ient  System 

Air  Force  Pampnlet 

Air  Force  Regulation 

Air  Refueling  Squadron 

Air  Training  Cor.iinand 

Bombardment  Squadron 
Bombardment  Wing 

Castle  Air  Force  Base  Regulation 
Combat  Crew  Training  School 
Cockpit  Procedures  Trainer 
Curriculuiii  Review  Board 
Curriculum  Review  Group 

Distinguished  Graduate 
Deputy  Commander  for  Operations 
Director  of  Training 
Aircrew  Scheduling:  Bomber 
Operations  Systems  Management  Branch 
Aircrew  Scheduling:  Tanker 
Standardization/Evaluation  Division 
Standardization/Evaluation:  Bombers 
Standard! zation/Eval nation:  Tankers 
Training  System  Management:  Bombers 
Training  System  Management:  Tankers 
Instructional  Systems  Development  Division 
Instructional  Systems  Jevelopi,ient  Division: 
Scientific 

Follow-On  Operational  Test  and  Evaluation 

General  Accounting  Office 

Strategic  Air  Commana  Headquarters 

Instructional  Systems  Uevelopiiient 

Mission  Accomplishment  Report 

Operational  Test  and  Evaluation 

Progress  Record 

Strategic  Air  Coiimafnl 
Standard izat ion/ Eva  1 uat ion 


^4 


TAG 

Tactical  Air  Coimnand 

TAR 

Training  Accompl  ishrnent  Report 

TRP 

Training  Review  Panel 

UNT 

Jndergraduate  Navigator  Training 

UPT 

Undergraduate  Pilot  Training 

WST 

Weapon  Syscem  Trainer 
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APPENDIX; 

KEY  DATA  ITEMS  AT  THE  93  BOMBARDMENT  WING 
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INITIAL  QUALIFICATION  CRITERION  TEST  RECORD 


CHI  TER  ION  TEST 


GRADE 


AIRBLEED 

(CB). 

AIR  REFUELING 

(CB)_ 

AUTOPILOT 

(CB). 

COMMUNICATIONS 

CM4(CB) 

CM2(CB) 

CREW  COORDINATION 

(CB). 

ELECTRICS 

(CB). 

EMERGENCY  PROCED. 

(BP). 

(CB) 

ENGIMS 

(CD) 

FLIGmT  CONTROLS 

(CD) 

!  UELS 

(CL) 

fU'DPAULlCS 

(CB) 

INSTHjf'iGIS 

(OB) 

life  SUri'uRT 

(CB) 

mu  lIVLL  LL9 

,3,7(CB) 

LL4(Cb) _ 


CHART  (OF,; 


■ijU\ 

'.AL  FiOiCED, 

CFT  E.'AL 

1 '  1  ■' 

N 

(CD) 

I’i  !.^ 

(CL) 

A  L  / '  * 

(OB) 

Min 

PiiAOF  TEST 

^  D) 

.  ■  'C 

T  [  ST 

(CD) 

INSTRUCTOR  INITI,'.; 


NA^!L 


CLASS 


2H 


CCTS  STUDENT  CRITIQUE 
(ACADEMICS  PHASE) 


TYPE  AIRCRAFT 

DATE 

COURSE  CODE 

CREW  POSITION 

CLASS 

BASE  ASSIGNED 

RANK 

TOTAL  FLYING  HOURS 

appreciate  you  rating  each  of  the  areas  listed  below  which  can  be  rated  Out¬ 
standing  (0),  Satisfactory  (S)  ,  or  Unsatisfactory  (U)  .  Check  (•/)  the  rating 
each  item  which  best  expresses  your  opinion.  If  you  rate  an  item  Unsatisfactory 
(U) ,  give  specific  comments  and  recommendations  for  improvement. 


ITEMS 


1.  INSTRUCTORS;  (Class  Control,  Attitude,  Enthusiasm, 
Helpfulness,  Understandable) 


2.  INDIVIDUAL  ASSISTANCE:  (Remedial  Instruction, 
Counseling) 


3.  TRAINING  METHODS:  (Amount  of  Theory  &  Practical, 
Use  of  training  time.  Student  Participation) 


4.  TRAINING  LITERATURE:  (Availability,  Use  and  Help¬ 
fulness,  Student  Study  Guides,  Workbooks,  Tech¬ 
nical  Orders) 

0.  VISUAL  AIDS:  (Availibility ,  Use  and  Helpfulness, 
Films,  Transparencies,  Charts) 

^  SYNTHETIC  TRAINING  AIDS:  (Pilot  Simulators,  T-40, 
F-10,  T-4,  T-IA  Trainers) 


7  EXAMINATIONS,  TESTS:  (Understandable,  Admin¬ 
istratis...,  Critique) 


COMMENTS 


RATINGS 


8  OVERALL  EVALUATION  OF  COURSE 


additional  COMJiENTS  (Use  reverse  side  if  more  space  if  required) 


CASTLE  ^ORM^  49  REVISED  ** 
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KC-i35  PROGRESS  REPORT  -  FLitiHT  liHASE  -  PILOT 


STUDENT  (Last 


SORTIE  DATE 


I,  First.  Ml): 


SORTIE  NUMBER: 


TAKEOFF  TIME: 


CREW  NO:  CREW  POSITION  (CIRCl£  0»C>: 

P  CP  PUP  P  REQ  IP  REO 


FLYING  TIME:  AIRCRAFT  NUMBER: 


INSTRUCTIONS:  Coluan  1:  snter  nuaber  of  tiaas  avant  accoaplishad.  Coluan  2:  anter  F  or  0  if  Faaillarization  or  Oa- 
aonstration  only.  Rsaalning  coluans:  antar  nuaber  of  tiaas  event  accoaplishad  at  each  proficiency  level  (DO-10  51-11). 


(kquirad  Instr  CBOCTTMTION  EXTENSIVE 

Intervention  (SQUIRED  ASST  iSQUllSO 


SUBSTANIAL 


LIMITED 


ASST  (EQUDCD  ^  (SQUIIO  TECHNIQUE 


OQfCHDC  fOi  NOE-ERRDRS 


OGRBYSMENT 
- 3,5 - 


Student 

Parforaanca 


LACK  OF 
KMMLEDGE 


INFLIGHT 


SIGNIF  ERRORS 
OR  DEVIATIONS 


SLIGHT  ERRORS 
OR  DEVIATIONS 


NO  ERRORS 
OR  DEVIATIONS 


□  WST  lo  Ql  PTOFICIENCY  LEVELS  {NARRATIVE  DESCRIPTION  OF  STUDENT  PERF0W4ANCE,  INCLUDING 
<  iV  |.0|l.5i2.0l2.5l5.0i3.5{4.0  DKOSPANCIES/REOMCftEO  CORRECTIVE  ACTION.  IF  fCCESSARY. 


TAKEOFF  AND  CLIMB 


40  PREFLIGHT 


PRETAKEOFF 


P08 I  TAKEOFF  _ 


PI  1  TAKEOFF  NIGHT _ 


P12  T.O.  CLIMB  PROC 


P20  T.O.  GYRO  MODE 


P25  T.O.  WET 


P35  SIM  ENG  FAIL,  TO  PONT 


P55  T.O.  HVYWT  30*  _ 


13  MITO  (LEAD) 


P24  NITO  (NOT  LEAD) _ 


P26  CELL  DEP/ JOIN-UP  (LO) 


P26  CELL  DEPT/JOINT-UP(N-IJD) 


102  INST  DEPT 


99  CLIMB 


P06  CREW  COORD 


■■■■■■■■■ 

■■■■■■■■■ 


AIR  REFUELING 


P22 

CELL  FORM 

14 

A/R  FOFM/TKR  CELL  (LO) 

R19 

A/R  FO»4/TKR  CELL  (N-LD) 

R36 

ENROUTE  RENDZ 

R4  2 

PRIMARY  RENDZ 

R4  0 

TKR  ALT  RENDZ 

R4  6 

TKR  REfOZ  OVRN  PROC 

R4  1 

TKR  A/R 

R44 

TKR  A/R  INOOC 

R15 

FTR  A/R  INDOC 

R4  5 

TKR  A/R  BREAKAWAY 

■II 

■Bi 


NAVIGATION 


N16 

NAV  AID  FIXING 

N51 

CELL  NAV  LEG 

P37 

AP  OFF  CRUISE 

79 

CRUISE 

P06 

CREW  COORD 

"  CODE 


TRAINING  EVENT 


PROFICIENCY  LEVELS  | NARRATIVE  DESCRIPTION  OF  STUDENT  PERFORMANCE ,  INCLUDING 
QISCREPACIES/ieDOI#C>CED  CORRECTIVE  ACTION.  IF  NECESSARY. 


II309S9333E5!S  I 


AIR  WORK 


P7  3  APP  TO  INIT  BUF  A  RECOV 


P57  SPOILER  DEMO _ 


P70  TRIM  DEMO _ 


P7  1  I  SIM  JAWCD  STAB  PROC 


P4  0 


P41  EMER  OP  MING  FLAPS 


P37  AIRWORK 


LANDINGS 


P14  ' 

LANDING  FULL  STOP 

P16 

LANDING  -  NIGHT 

P33 

LND  30'FLAP 

P61 

APP  &  LND  SIM  3  ENG 

P97 

TOUCH  &  GO  LND 

PI  3 

LANDINGS  (TOTAL) 

19 

CROSSWINO  LOG 

P06 

CREW  COORD 

81 


INSTRUMENTS 


HOLDING 


DESCENT 


ENROUTE  DESCENT 


PEN  (PUBLISHED) 
ABN  RADAR  APP 


PAR  APP 


ILS  APP 


ILS  AUTO  APP 


ILS-GYRO  MODE 


PREC  APP 


TACAN/VOR/LOC  _ 

ASR  APP 


NON-PREC  APP 


MISSED  APP 


INST  APP  &  M/A  SIM  3  ENG 


AP  &  G/A  SIM  3E  (RD  R4R  OFR 


VOR/TACAN  PROC 


CREW  COORD 


88! 


881 


MISCELLANEOUS 


G18  BAILOUT/DITCH/CRASH  LND 
P03  MISSION  PREP 


P04  ACFT  EQUIP  FAM 


INS 


N68  INS/DNS 


P05  CHECKLIST  PROC/USE 


POSTFLIGHT 


P31  INST  TECHNIQUE 


39  MSN  PAPERWORK 


STUDENT  SIGNATURE: 

INSTRUCTOR  SIGNATURE: 

KPCH: 

VER: 

31 


EMM  FIOC  I  LAHPUICS  I _ IHSTKUHEHTS _ I  BOMBING  WPW  SYS|  MAVIGATIOH  I  AIR  REFUELING  I  TAKEOFF/CLim  |  PROF 


B-52  piuTT/oopiLcrr  sncEin:  training 


CREH  NO. 


ITEM#  DESCRIPTION 


SOlO  SORTIE 


MISSION  PLAN 


OORDINAT 


TBii533^ 


TAKEOFF  CP  DUTIES 


TO  A  CLIMB  PROC 


TAKEOFF  met  ■ 


TAKEOFF  NIGHT 


M  1  TO 


CELL  DEPT  i  JOIN 


■■■■■■■■■■■■■■■■■■■■■■■I 

■■■■■■■■■■■■■■■■■■■■■■■I 


■■sB&SaS8S8|8BS&i 

SisssaMsssss! 


AR  TANKER  A/P  OFF 


AR  RCVR  A/P  OFF 


i^mgaminogggggaigggaiaSgBB 

JHHBiaaaggiHaaaaaaggg 


NIGHT  IMT  TA/EVS  NAV  L£ 


MULTI  RELEASE  LOW 


■Nt^.irNnrNg.BBN1MNM,N.NMMN— ■■■■■ 


HI  SPEED  BOMB  RUN 


INSTRUMENT  DEPT 


IliiLfl 


PAR  APPROACH 


umimmmmmmm 

isaaaffii _ 

iRiaaBaiiiaaBBi 

iBBmaaBaaBBHi 


ABN  RADAR 


IffTl 


SIM  A  EIC  AfP  i  00  AR 


SIM  S  ENG  ARP  S  IN 


CASTLE  I’*  REVISED  •  • 

OCT  S3 


f 

eCTS  STUDENT  CRITIQUE 
•  (Flying  Phase) 

TYPE  AIRCRAFT 

DATE 

COlfRsB^  CODE 

CREW  POSITION 

CLASS 

BASE  ASSIGNED 

TAME 

I 

RANK 

total  flying  HOURS 

INSTRUCTIONS:  To  assist  us  in  improving  the  B-52/KC-135  Combat  Crew  Training  cours'  we 


would  appreciate  you  rating  each  of  the  areas  listed  below  which  can  be  rated 
Outstanding  (0),  Satisfactory  (S) ,  or  Unsatisfactory  (U) .  Check  the  rating 

for  each  item  which  best  expresses  your  opinion.  If  you  rate  an  it<  m  Uni^atij- 
factory  (U)  ,  jivc  specific  comments  and  recommendations  for  imj.  rovcirent . 


\  ITEMS 

COMMENTS 

- i 

R/i.TING  1 

i 

D 

a 

a 

i  1.  INS'.  ;'ICTCR3  :  (Attitude,  Enthusiasm, 

1  inebs ,  Understandability) 

1 

1 

.  IjINJ  TRAINING;  (Use  of  time,  equip- 
(  nent,  maintenance,  flying  schedule) 

1 

1 

;  3.  SYNTHETIC  TRAINING  AIDSr"- (Pilot  sim¬ 

ulators,  TvlO,  F-lA  Tt4ihers) 

1 

4.  STANDARDIZATION  DIVISION:  (fests , 
Flying  evaluation,  critique) 

1 

1 

S.  OVERALL  EVALUATION  OF  FLYING 

TRAINING 

1 

1 

1 

1 

ADDITIONAL  COMMENTS  (Use  reverse  if  more 

space  is  required) 

I 

i 


JVPR*a^2  revised 
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DATE  COMPLETCO 


CERTIFICATE  OF  AIRCREW  QUALIFICATION 


EXAMINEE  IDENTIFICATION 


NAMK  (Last.  First,  Middle  Initial) 


ORSANIZATION  AND  LOCATION 


ACFT/CNEW  POSITION  I  tLIOISILITV  PERIOD 


QUALIFICATION 


GROUND  phase 


FLIGHT  phase 


EXAMINATION /CHECK 


QUALIFICATION  LEVEL 


QUALIF  lED 


UNQUALIF  lEO 


RESTRICTION 

(Explain  in  DUE  dates 

Comments) 

□  ve«  □  NO 


additional  TRAINING 


CXriRATION  OATC  OF  QUALIFICATION 


COMMCNTS  (If  more  ipQ<  e  is  needed,  continue  on  reverse) 


DATE  ADDITIONAL  TRAINING  COMFLCTED 


CERTIFICATION 


TYPE  NAME  AND  GRADE 


flight  examiner 


REVIEWING  OFFICER 


final  affroving  officer 


ORGANIXATION 


I  CFR  fit  Y  that  I  have  been  briefed  and  understand  the  action  being  taken  this  date 


M.  ^  FORM  - 

AF  ...  „  8 


TYFSO  NAME  AND  GRADE  OF  EXAMINEE 


FREVIOUE  EDITION  WILL  EE  USED 


^{.5.  C.  I9Ci4-42C*979/I33?9 


reply  to 

ATTN  OF:  DOV 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  930  BOMBARDMENT  WING  (SAC) 
CASTLE  AIR  FORCE  BASE,  CA  95342 


SUBJECT: 


Stan  Eval  Report  for 


fo  93  BMW/DO- 1  93  BMW/DO-2 

1.  Stan  Eval  completion  date: 

2.  The  results  of  the  initial  qual i f icat ion/requal i f icat ion  standardiza¬ 
tion  checks  are  submitted  below.  Annual  rates  are  based  on  initial  qual¬ 
ification  checks  only. 

3.  Summary  (by  crew  position): 

a.  This  class:  CHECKED  ^  ^  ^  %Q1  Xqi 


PILOT 
COPILOT 
RADAR  NAV 
NAV 
EWO 

GUNNER 

TOTAI.S 


b.  Annual  Classes:  to 


PILOT 
COPILOT 
RADAR  NAV 
NAV 
EWO 

GUNNERS 

TOTALS 


c.  Requalification 

PILOT  _ 

PILOT  CPT  (Only)  _ 

RADAR  NAV/ NAV  _ 

EWO  _ 

TOTALS 


JOHN  R.  WRIGLEY,  Major,  USAF  2  Atch 

Chief,  Stan  Eval  Bomber  Branch  1.  CAFB  Form  7A 

2.  Reasons  for  Student 
Q2/Q3 


P  VlJi  <■  ,  .  .  .  I  S  -III  r  /’  f./  ,  I  ,,n 

36 


DOV  FL  #12 


B-52  RADAR  NAV/NAV  TRENDS  ANALYSIS  WORKSHEET 


EVALUATOR: 


MISSION  PLANNING 


PUBLICATIONS 


22 

PREFLIGHT 


23 

PRETAKEOFF 


24 

TAKEOFF 


25 

CLIMB 


26 

LEVEL-OFF 


27 

CRUISE 


EMERGENCY  PROCD. 


COMMUNICATIONS 


CREW  COORDINATION 


DESCENT  &  LANDING 


33 

POSTFLIGHT 


35 

AIR  REFUELING 


36 

BOMBING 


GEN.  NAVIGATION 


CEL.  NAVIGATION 


DATE  EVALUATED: 


POSITION: 


TYPE  CHECK: 


N/0  I  U/1 I  T/2 


REMARKS 


"emergency  print;  eaqjires  29  Apr  86" 


K.\DAR  NAV-'NAV  TRENDS  ANALYSIS  WORKSHEET 


AR 

EA 

37 

LOW  LEVEL 

NAV  . 

46 

EQUIPME 

NT 

OPERAT. 

41 

GUIDED 

a; 

R  MISSIL. 

43 

TERFUXIN 

AVO I  DANCE 

47 

JUDGEM . 

& 

COMPL. 

42 

INSTRLC 

'P; 

R  CHECK 

42 

QL’AL  I  F  1 

■-A 

TION  EXAM 

L’/'l  T/:|  (1/3  1  UP4 


REMARKS 


INSTR  t 


Key  Punched 


Exams:  95-100-  4  85-89=  2 

90-94  «  3  0  -84-  1 


INSTRUCTOir’S  NAME: 


rv  ALUA  lOR'S  NAME.. 


ACADEMIC  EVACUATION  WORKSHEET 


RGANIZATION: 


OVLKALL  I'F:  kFOKMANCE  ; 


LESSON  CODE; 


□  EXCELLENT 

□  SATISFACTORY 


a  MARGINAL 

UNSATISFACTOP.V 


CVALUATOf?  instructions.  Thii  lorm  it  d«ti9n«d  to  pre«»d«  on  •voIw^iIoa  oI  •ca^omlc  ln«trwc«»T  Id*  !••••#«  el 

lAttfwctiMi.  Pri«r  to  ovotuotion,  the  owoluotot  OMJtt  rowiow  tK#  court#  book,  intiructor  guid#  nnd  tb*  litt  of  oducetloo  ond  trolnln^ 
ro«u«r#montt  (ETR't)  tot  Utfton  to  b#  moniterod.  Th#  obiocNvo#  mutt  bo  mot.  tfio  Inttruclor  •uido  oiutt  bo  foHowod.  ond  oil  CTR't 
mutt  bo  cevorod.  TK#  9rad#  will  bo  oiiboi  YES  oi  NO.  For  oH  oroot  morbod  NO.  oloooo  ojrplotn  IuIIt  in  fbo  romotbt  toctlon. 
tho  ontirt  iotton.  it  on#  ot  tbogrodod  ftwb<-or«ot  doo«  net  opply  Rut  H/'A  under  ibo  YES  cpluion.  A  frodo  of  YES  IndIcotot  tbot  tbo 
•  voluotod  Item  moott  ttendeidt  conductvo  to  pood  tfudont  looming.  A  prod#  NO  by  onf  OPtoHtJipd  Itpm  will  nocpttlfoto  on  untolit* 
tectorv  9«odo  m  evoroll  portormonct.  It,  in  the  opinion  of  tn#  oweluotor  the  lotton  wot  not  up  to  tbo  collbor  conducive  to  tiudont 
itammg,  th#  ovoiueter  mutt  owotd  on  overall  prod#  of  UntotItfoctofY  ovon  tbeugb  no  otlorftliod  oroot  wore  prodod  NO. 


SECTION  I  method  Of  instruction-  {  j  tocH  typo  of  modio  utod  end  compiotp  tno  cortotponding  cboekUttt.) 


RTT.  Cf  T.  CPT,  SIMS.  WS7 


Audio-' Vi  tuul  *■*  Sound'Sltdt,  Vld<oropt 


Lttton  f«Rt 


field  T rip 


SfcCTlOH  11 


GRADING  AREAS  CHECKLIST 


A.  Ltrtuft  or  Swn'nQi 


ES  1  NO 


I  CiottruEin  oputorenct.  no  di ttrocfipn t»  lipbfitig.  comforteblo  tomporoturt 
4.  rfOiiHii  ;  Mntvnoit  -  Q«oi(obit,  pfoebocbtd.  current,  offoerivo 
3  Oroeni inflow  atronnon.  motivotion,  overview,  toeuonco.  review 
4.  Ploiiorn'  Ci'^ifoiton  t*<c  f  -  oppeorwicc,  monnontmt,  eenfioonco 
*'  ^oocbit  ^  Tr<n'iiOwtt  -  ttwdont  cvnforod.  mtorot*  level,  (iutt  control 
f  LvttM'  hnj  j‘»t«i*tni<  -  ETR'»  co  ytrod.  obtocfivot  mot 
V  Ti'ne  Control  -  Siort  ond  end,  oMocotion 

6  volwotio''  -  qutttioning  focHniqutt.  grodmg.  crtfonon  toott  uttd 


B.  T  rginr 


I.  Condition  -  citon,  ovo'lobit.  working  order,  curronf,  roolitn* 

7.  I  •tiructnr  Owide  --  curroncr.  pdoq.i#<y 

lioi^Miig  Mcitonolf  -  ovoHoblt,  Current,  effective 
4.  Inrreduction  -  pre^bnef,  ottentlen.  mefivelion.  review,  preview 
Ie.  Inttructnr  Cburectentfict  *  oppooronce.  monnoHtmt.  confidenct 
c.  Inttructnr  Knowledge  -  fi.oteriel,  treinoi.  centol# 

Orgoniietion  *  tepwence,  trontitiont,  fnterott  lovot 
0.  Lv««oe«  Ron<'*rementt  «  ebftctivet  met.  CTR*e  covered 
Timv  Cuntrni  ttnrt  onrf  end.  nllecptlen 
to  Evninutien  «vci>niquct.  ftodlnp.  rritenon  fettt  utod 
tt.  Conriution  •  tunimriry,  ottipnmont.  romptl voHon 


castle 
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C.  Awdto'-Vitwol  ~ 


ritu 


1.  Ifttw*  Hoint  ■*  o«otiobi«,  knowl«d9«obl««  ipf»«oranc«.  c«i*rt*vu« 

7  Lertfiuog  Ar«o  *  nppcoronr*,  dtt*ra;tlont.  lighllng^  t«ffiip*»o«ur« 

i  Equ<f«m*4«f  owoilob)*,  deon,  opwfotmq  condition 

4.  CiMtiitot  -  Jirvv  iiofrotioft.  wi*t»olft.  poc««  corfont,  ln<or*«t 

5  Orgoni»oii«>  *  ottontuM.,  mofivoiiuit,  nwor^icw,  row*o«i,  of (oc ti von«a » 

6  E«oiuotion  -  e«ofcitc«,  cvitono^  icata 

Lotaon  Roqui»oi^ent«  >  CTR'a  eovorod»  obioctivo  mot 
8.  Inatrwctor  ->  eveiloblo,  k nowiodgooblo 


D.  L  0  aaon  1 ci t 


1.  ffoining  Moter>o(a  -  ovoiiuble,  curre*>t,  offoclivo,  occutoto 

2.  Loaaon  Roquiromonta  —  ETR's  covorod«  ob)#ctivoa  mot 

3.  Evoiuotion  ••  criterion  te»t  uaod.  orndod,  diacuiood 
*1.  Instrvctor  •<  ovoiloblo,  ViiowlodgooLlo 


L .  F lold  f  t 


r»  unkport'ito'' 


•  f.  ro*‘dMiri' 


2.  Tt(i«ninq  Mo*mf>l5  --  ovoiioble,  tonditmn 
1.  Troinii<g  Cond't'nna  d' atror^iona,  lighting,  t#mporuti>ro 
4.  1  f  -rnniquoft  -  a*udent  centered  tntoroat  to«ol.  rlu»«  corttoi 

b.  Le«>  1  Reg  .iron.onts  •  £TR‘a  covoied.  ob|Oettvoa  mot 
^  I  e  -  a»o*t  ond  one.  o'locot»oe 
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SCCIU-H  ii: 


REMARKS  RECOmmEHOaTIONS 


signature  Oi  t  vaLU.  1  H 


signature  of  INSTRUCTOR: 


SE  CTION  IV 


CORRECTIVE  action 


^  I  NONF  Rfc  QUIRE  D  I S<9f'o*b're  in  Sot  non  I V  i »  rogutrod  only  If  thoro  la  o  grodo  of  UNSAT*) 


DATE 


SIGNATURE  OF  SO  OR  DIVISION  COMMANDERi  SIGNATURE  OF  CRB  CHAIRMAN. 


DEPARTMENT  OF  THE  AIR  FORCE 

HEADQUARTERS  930  BOMBARDMENT  WING  (SAC) 
CASTLE  AIR  FORCE  BASE,  CA  95342 


REPLY  TO 
ATTN  OF:  DOV 

SUBJECT:  Training  Device  Evaluation 

TO:  93  BMW/DO-2 
93  BMW/DOD 
93  BMW/DOT 
93  BMW/MAAD 
93  BMW/DOV(B/T) 

IN  TURN 

1.  The  following  training  syllabus  was  evaluated  on  _ . 

a.  Training  Device  _ .  Serial  Number  _ . 

b.  Type  of  mission  _ . 

2.  The  following  areas  were  found  to  be:  SAT  UNSAT 

a.  Quality  of  instruction:  _  _ 

b.  Lesson  technical  accuracy  _  _ 

c.  Lesson  Content  _  _ 

d.  Training  Material  Quality  _  _ 

3.  Remarks: 


4.  The  next  evaluation  will  be  made  not  later  than  _ 

(date) 


(Evaluator*s  Signature) 


Evaluator's  printed  name  and  rank 

Peace.  .  ,  .  i  s  our  Profession 
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RECOlWENDATION  FOR  TRAINING  IMPROVEMENT 


OtflU  retm  211  (Hmm)  MM  M 
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